You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 09:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 16, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21099210679,9sleep30-21swapper/319:06:493
2104029381,8sleep10-21swapper/119:06:041
2105428068,8sleep00-21swapper/019:06:150
1921327666,6sleep20-21swapper/219:05:072
21441993331,1cyclictest14321-21diskmemload23:48:283
2143999290,4cyclictest0-21swapper/119:15:141
21441992826,1cyclictest15706-21if_enp2s021:10:203
2143999280,2cyclictest4808-21date23:00:011
2143999280,2cyclictest4808-21date23:00:001
34352260,2sleep23244-21kworker/2:123:21:442
2144099260,26cyclictest0-21swapper/222:44:572
2143999261,22cyclictest0-21swapper/123:07:331
2144099250,25cyclictest0-21swapper/223:25:442
2143999254,1cyclictest14321-21diskmemload22:16:081
2143999240,1cyclictest0-21swapper/120:20:001
2143599240,14cyclictest32089-21sensors22:55:230
2143599240,14cyclictest32089-21sensors22:55:230
21440992315,7cyclictest37850irq/131-snd_hda23:05:172
2143999233,2cyclictest25728-21ssh22:05:121
2143999231,3cyclictest14321-21diskmemload22:03:041
2143999231,1cyclictest0-21swapper/123:20:271
2143999230,3cyclictest0-21swapper/123:45:291
2143999230,1cyclictest0-21swapper/100:15:221
2143999223,1cyclictest14321-21diskmemload23:55:271
2143999220,4cyclictest0-21swapper/121:40:141
2143999220,16cyclictest0-21swapper/122:50:141
21435992219,2cyclictest49-21kcompactd022:15:150
2143599220,20cyclictest0-21swapper/023:59:220
2144199212,15cyclictest0-21swapper/323:33:043
21441992118,2cyclictest22632-21cron20:20:003
21440992119,1cyclictest32-21ksoftirqd/222:47:412
2144099210,21cyclictest0-21swapper/223:36:152
2143999213,1cyclictest14321-21diskmemload21:22:071
2143999210,4cyclictest13064-21sh19:59:591
2143999210,1cyclictest0-21swapper/121:52:161
2143999210,1cyclictest0-21swapper/100:10:161
46662200,0sleep00-21swapper/000:39:080
2144199204,15cyclictest24455-21ntp_states21:40:233
21441992019,1cyclictest0-21swapper/319:55:123
2144099200,20cyclictest0-21swapper/223:12:182
2144099200,20cyclictest0-21swapper/222:32:232
2144099200,10cyclictest0-21swapper/200:07:142
2143999202,1cyclictest28131-21apache_processe22:30:131
2143999202,18cyclictest0-21swapper/120:10:251
2143999202,17cyclictest17872-21cron20:09:591
2143999201,1cyclictest22105-21grep21:15:211
2143999200,4cyclictest14321-21diskmemload21:31:331
2144199194,8cyclictest448-21apt-get00:35:153
21440991915,1cyclictest0-21swapper/223:16:262
2144099190,19cyclictest0-21swapper/221:41:562
2143999192,2cyclictest14321-21diskmemload00:08:051
2143999192,2cyclictest0-21swapper/123:30:101
2143999192,16cyclictest0-21swapper/122:35:161
2143999191,1cyclictest14321-21diskmemload00:35:571
2143999190,2cyclictest0-21swapper/122:46:091
2144199185,8cyclictest5329-21ssh23:00:133
2144099187,2cyclictest1291-21runrttasks22:11:362
21440991818,0cyclictest0-21swapper/221:19:342
21440991815,2cyclictest32-21ksoftirqd/222:40:002
2143999188,5cyclictest6910-21ssh23:25:031
2143999187,6cyclictest17767-21ssh22:21:491
2143999181,2cyclictest0-21swapper/123:00:201
2143999180,17cyclictest26077-21head20:25:171
2143999180,13cyclictest4673-21ntpq19:40:221
2144199170,0cyclictest0-21swapper/321:21:183
21440991716,1cyclictest0-21swapper/221:38:512
21440991715,1cyclictest32-21ksoftirqd/222:18:032
2143999178,5cyclictest2161-21rm21:47:581
2143999177,5cyclictest9554-21ssh22:40:071
2143999173,1cyclictest0-21swapper/123:53:241
2143999173,13cyclictest0-21swapper/121:35:141
2143999170,1cyclictest0-21swapper/123:40:291
2143999170,1cyclictest0-21swapper/120:20:131
2143999170,0cyclictest0-21swapper/100:30:231
2143599173,7cyclictest27698-21perf22:30:000
21440991612,3cyclictest15941-21ls00:20:012
2144099161,14cyclictest1302-21ssh21:47:072
2143999169,6cyclictest15528-21grep21:10:181
2143999168,4cyclictest21260-21sh23:10:471
2143999167,5cyclictest27537-21sh23:39:461
2143999166,9cyclictest20116-21ssh00:22:431
2143999163,3cyclictest21877-21apt-get00:25:041
2143999162,5cyclictest1546-21ntp_states22:10:221
2143999161,3cyclictest1120-21strings20:40:201
2143999160,3cyclictest0-21swapper/119:54:591
2143999160,1cyclictest26695-21sh23:15:201
21435991614,2cyclictest0-21swapper/021:11:350
2143599160,0cyclictest0-21swapper/000:20:350
2144099157,5cyclictest8918-21apache200:02:542
2144099150,1cyclictest14321-21diskmemload22:50:182
2143999158,6cyclictest6750-21basename19:45:181
2143999158,6cyclictest16806-21rm21:58:281
2143999157,4cyclictest4371-21taskset20:48:211
2143999152,3cyclictest29323-21sh00:04:481
21441991410,2cyclictest21923-21tr19:10:143
21441991410,2cyclictest21923-21tr19:10:143
21440991412,1cyclictest32-21ksoftirqd/223:03:302
21440991411,2cyclictest1866-21grep19:35:172
21440991411,2cyclictest10100-21df21:00:172
2144099140,14cyclictest0-21swapper/222:58:102
2144099140,14cyclictest0-21swapper/222:58:092
2144099140,14cyclictest0-21swapper/221:53:382
2143999148,5cyclictest31848-21taskset20:36:091
2143999148,5cyclictest2932-21taskset19:35:571
2143999148,5cyclictest151872sleep120:01:411
2143999147,6cyclictest4158-21ssh21:25:421
2143999140,2cyclictest7767-21grep20:55:161
2143999140,2cyclictest0-21swapper/122:29:281
2143999140,1cyclictest0-21swapper/119:30:141
2143599148,4cyclictest11881-21apt-get21:55:170
2143599141,6cyclictest11749-21basename21:05:000
21435991410,3cyclictest11611-21grep22:40:280
145542140,0chrt0-21swapper/300:18:193
2144199139,3cyclictest29327-21sed00:30:283
2144199139,3cyclictest25369-21grep20:25:143
2144199139,3cyclictest23241-21ssh00:00:063
2144199139,3cyclictest15023-21grep20:00:273
2144199137,4cyclictest31762-21needreboot22:55:203
2144199137,4cyclictest31762-21needreboot22:55:203
2144199133,5cyclictest224-21systemd-journal19:20:003
2144199133,2cyclictest224-21systemd-journal21:49:593
21441991313,0cyclictest0-21swapper/323:14:073
21441991310,2cyclictest9143-21grep22:15:283
21441991310,2cyclictest30164-21grep22:55:003
21441991310,2cyclictest29932-21sed22:30:263
21441991310,2cyclictest18868-21tail20:10:203
2144199130,9cyclictest0-21swapper/320:20:123
2144099139,3cyclictest27540-21tail19:20:212
2144099139,3cyclictest25107-21wc19:15:202
2144099137,2cyclictest32-21ksoftirqd/223:56:072
21440991311,2cyclictest32-21ksoftirqd/221:00:012
21440991310,2cyclictest32305-21head19:30:212
21440991310,2cyclictest27822-21cat22:30:002
21440991310,2cyclictest10615-21head23:50:172
2144099130,13cyclictest0-21swapper/222:08:362
2143999138,4cyclictest9867-21cut21:00:161
2143999138,4cyclictest27629-21seq20:30:011
2143999138,4cyclictest22439-21head19:10:171
2143999138,4cyclictest22439-21head19:10:161
2143999137,5cyclictest13033-21cat21:05:161
2143999137,5cyclictest13033-21cat21:05:161
2143999131,4cyclictest26844-21sshd19:20:171
2143999130,3cyclictest224-21systemd-journal19:25:001
2143599139,2cyclictest887-21basename19:35:000
2143599139,2cyclictest20754-21tr20:15:150
2143599138,4cyclictest7744-21snmpd20:30:580
2143599138,4cyclictest2683-21apache_volume22:35:150
2143599136,6cyclictest30101-21munin-run22:55:000
2143599135,6cyclictest27176-21munin-run00:30:000
2143599134,1cyclictest0-21swapper/000:16:040
2143599132,1cyclictest141rcu_preempt22:23:070
2143599131,2cyclictest0-21swapper/022:03:560
21435991310,2cyclictest794-21df22:10:170
21435991310,2cyclictest7404-21cat19:45:260
21435991310,2cyclictest5201-21sed19:40:270
21435991310,2cyclictest4237-21cron21:49:590
21435991310,2cyclictest29944-21sed22:30:260
21435991310,2cyclictest2693-21tr20:45:160
21435991310,2cyclictest20322-21wc20:15:000
21435991310,2cyclictest14988-21sed23:30:150
21435991310,2cyclictest13-21ksoftirqd/021:21:530
2144199129,2cyclictest9363-21tr21:30:173
2144199129,2cyclictest9361-21tr21:00:013
2144199129,2cyclictest8979-21cut19:50:173
2144199129,2cyclictest7405-21grep23:25:163
2144199129,2cyclictest6113-21wc21:50:233
2144199129,2cyclictest4039-21grep20:45:283
2144199129,2cyclictest29056-21head23:40:183
2144199129,2cyclictest26938-21ssh23:15:203
2144199129,2cyclictest22072-21grep23:35:193
2144199129,2cyclictest16105-21wc22:20:253
2144199129,2cyclictest1139-21ssh22:10:203
2144199128,3cyclictest5677-21awk19:45:013
2144199128,2cyclictest10073-21tr23:50:143
2144199120,3cyclictest0-21swapper/321:55:173
2144199120,11cyclictest16375-21apache_accesses00:20:123
2144099129,2cyclictest30435-21grep19:25:272
2144099129,2cyclictest28526-21sort21:20:182
2144099129,2cyclictest27908-21tr20:30:122
2144099129,2cyclictest23746-21chrt22:03:272
2144099129,2cyclictest22361-21cut19:10:172
2144099129,2cyclictest22361-21cut19:10:162
2144099129,2cyclictest22224-21head20:15:262
2144099129,2cyclictest21866-21cron00:25:002
2144099128,2cyclictest5161-21cron00:40:002
2144099126,5cyclictest32-21ksoftirqd/221:25:162
2144099125,6cyclictest0-21swapper/223:30:222
2144099124,1cyclictest141rcu_preempt21:55:262
21440991211,1cyclictest37850irq/131-snd_hda21:12:582
21440991210,2cyclictest10207-21kworker/u8:020:28:152
21440991210,1cyclictest32-21ksoftirqd/220:10:192
2144099120,9cyclictest0-21swapper/221:30:142
2144099120,11cyclictest4662-21grep23:45:272
2143999125,6cyclictest6569-21taskset20:51:081
2143599129,2cyclictest25052-21sed21:40:280
2143599129,2cyclictest23282-21grep19:10:260
2143599129,2cyclictest23282-21grep19:10:260
2143599129,2cyclictest10485-21basename19:54:590
2143599129,2cyclictest10308-21uname21:30:220
2143599128,3cyclictest27383-21ls21:20:000
2143599127,1cyclictest13-21ksoftirqd/023:29:590
2143599120,2cyclictest0-21swapper/021:38:190
2144199119,1cyclictest17935-21sed23:55:223
2144199118,2cyclictest590-21expr23:20:163
2144199118,2cyclictest4924-21egrep20:50:183
2144199118,2cyclictest4216-21grep22:35:253
2144199118,2cyclictest3982-21cut00:10:173
2144199118,2cyclictest31746-21grep19:30:163
2144199118,2cyclictest26399-21grep22:05:153
2144199118,2cyclictest23497-21ssh22:25:323
2144199118,2cyclictest23400-21ssh21:15:363
2144199118,2cyclictest23307-21ssh22:03:023
2144199118,2cyclictest21503-21ssh22:48:253
2144199118,2cyclictest1875-21head19:35:173
2144199118,2cyclictest1760-21ssh21:25:023
2144199118,2cyclictest14685-21ssh23:06:053
2144199118,2cyclictest11179-21wc21:00:273
2144199118,2cyclictest10438-21uniq22:40:183
2144199116,4cyclictest496-21dbus-daemon20:06:573
2144199112,5cyclictest8025-21cron19:49:593
2144199110,10cyclictest23204-21grep00:25:173
2144199110,10cyclictest12747-21df21:05:153
2144199110,10cyclictest12747-21df21:05:143
2144099118,2cyclictest4657-21wc20:50:002
2144099118,2cyclictest32109-21taskset20:39:292
2144099118,2cyclictest31488-21ssh23:41:452
2144099118,2cyclictest29162-21sort00:30:272
2144099118,2cyclictest15341-21if_enp0s31f622:20:182
2144099118,2cyclictest12947-21grep21:05:172
2144099118,2cyclictest12947-21grep21:05:162
2144099118,2cyclictest12616-21tail19:55:252
2143599119,2cyclictest0-21swapper/000:05:140
2143599119,1cyclictest28730-21chrt00:04:020
2143599118,2cyclictest8086-21uname20:55:180
2143599118,2cyclictest7566-21ssh00:12:300
2143599118,2cyclictest5779-21ls23:00:160
2143599118,2cyclictest3295-21cut21:25:220
2143599118,2cyclictest29630-21ssh00:30:440
2143599118,2cyclictest28735-21cut19:25:010
2143599118,2cyclictest2582-21ssh23:45:030
2143599118,2cyclictest25710-21ssh22:05:110
2143599118,2cyclictest24157-21ssh23:36:240
2143599118,2cyclictest20883-21tail23:10:280
2143599118,2cyclictest12412-21grep23:05:170
2143599118,2cyclictest11245-21wc19:55:140
2143599118,2cyclictest1030-21grep23:20:190
2143599110,2cyclictest0-21swapper/023:15:020
2143599110,10cyclictest6171-21ssh21:50:220
2143599110,10cyclictest31826-21taskset20:35:520
2144199108,1cyclictest305982sleep319:25:523
2144199108,1cyclictest30047-21aten2_r7power_p00:05:163
2144199108,1cyclictest27654-21basename20:30:093
2144199108,1cyclictest16816-21tune2fs21:35:163
2144199108,1cyclictest1446-21sort20:40:263
2144199107,2cyclictest31666-21grep20:35:293
2144199107,2cyclictest26738-21tr19:20:153
2144099108,1cyclictest8499-21sort19:50:152
2144099108,1cyclictest627-21cut20:40:172
2144099108,1cyclictest5003-21sh00:10:222
2144099108,1cyclictest4143-21forks19:40:182
2144099108,1cyclictest23034-21aten2_r7power_v20:20:142
2144099108,1cyclictest15742-21aten2_r7power_c20:05:172
2144099107,2cyclictest7952-21chrt19:49:072
2144099107,2cyclictest6886-21idleruntime-cro20:55:002
2144099107,2cyclictest23074-21ssh00:25:162
2144099107,2cyclictest14447-21cat20:00:222
2143599108,1cyclictest308502sleep019:29:060
2143599108,1cyclictest28084-21ssh23:40:050
2143599108,1cyclictest13827-21expr20:00:160
2143599108,1cyclictest1334-21grep19:35:140
2143599108,1cyclictest10687-21ssh23:50:170
2143599107,2cyclictest4778-21chrt20:50:170
2143599107,2cyclictest25954-21sort20:25:170
2143599107,2cyclictest24826-21cut19:15:170
2143599107,2cyclictest23121-21tr20:20:140
2143599107,2cyclictest18339-21sed22:45:270
2143599107,2cyclictest15938-21cat20:05:170
2143599107,2cyclictest1476-21cat20:40:270
2143599107,2cyclictest14350-21cat21:10:000
2143599107,2cyclictest14350-21cat21:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional