You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 26, 2024 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10448210242,9sleep30-21swapper/319:09:483
10238210287,10sleep10-21swapper/119:07:061
1018528866,16sleep00-21swapper/019:06:250
1031828767,16sleep20-21swapper/219:08:062
1057799331,31cyclictest431-21taskset21:30:223
1057799310,30cyclictest0-21swapper/323:19:153
1057799310,2cyclictest3290-21diskmemload23:57:463
1057799310,1cyclictest0-21swapper/323:37:393
1057799301,1cyclictest30673-21chrt23:49:093
1057799300,2cyclictest0-21swapper/319:35:123
1057799300,29cyclictest0-21swapper/321:15:223
1057799300,1cyclictest0-21swapper/321:47:453
1057799290,28cyclictest0-21swapper/323:34:413
1057799290,28cyclictest0-21swapper/322:50:143
1057799280,2cyclictest3290-21diskmemload21:09:593
1057799280,2cyclictest0-21swapper/323:10:123
1057799280,25cyclictest0-21swapper/322:40:473
1057599280,27cyclictest0-21swapper/223:14:242
1057799274,22cyclictest3290-21diskmemload22:57:443
1057799270,5cyclictest3290-21diskmemload22:11:063
1057799270,26cyclictest0-21swapper/321:41:043
1057799264,21cyclictest3290-21diskmemload22:17:183
1057799261,24cyclictest3290-21diskmemload23:04:113
1057799260,2cyclictest23975-31su00:07:513
1057799260,25cyclictest0-21swapper/322:35:113
1057799260,25cyclictest0-21swapper/322:05:133
1057799260,25cyclictest0-21swapper/322:05:123
1057799260,1cyclictest0-21swapper/321:55:123
1057799260,0cyclictest0-21swapper/320:40:113
10575992624,1cyclictest0-21swapper/223:54:532
1057799252,22cyclictest0-21swapper/322:27:273
1057799251,23cyclictest0-21swapper/323:20:273
1057799251,1cyclictest3290-21diskmemload23:42:223
1057799250,1cyclictest6693-21/usr/sbin/munin22:45:193
1057799250,1cyclictest0-21swapper/321:35:463
1057599250,25cyclictest0-21swapper/221:48:592
1057799242,21cyclictest28520-21cat21:50:273
1057799241,1cyclictest31655-21grep19:50:323
1057799240,23cyclictest14772-21apt-key22:05:013
1057799240,23cyclictest14772-21apt-key22:05:003
1057799240,23cyclictest0-21swapper/323:05:103
1057799240,22cyclictest9322-21awk20:15:003
1057799240,22cyclictest9322-21awk20:15:003
1057799240,21cyclictest0-21swapper/321:26:493
1057599240,24cyclictest0-21swapper/221:13:342
1056799240,1cyclictest141rcu_preempt22:11:070
1057799232,20cyclictest20616-21aten2_r7power_p19:30:133
1057799230,22cyclictest31072-21sh23:25:283
1057799230,22cyclictest0-21swapper/322:30:453
1057799230,21cyclictest0-21swapper/300:10:103
1057799230,1cyclictest0-21swapper/319:15:293
1057799230,1cyclictest0-21swapper/300:16:363
1057099230,18cyclictest0-21swapper/121:26:341
1057799228,10cyclictest15281-21cut20:25:203
1057799223,18cyclictest28832-21apt-get20:55:133
1057799221,20cyclictest0-21swapper/321:20:143
1057799221,1cyclictest5982-21/usr/sbin/munin00:20:163
1057799221,1cyclictest3290-21diskmemload22:23:503
1057799220,21cyclictest0-21swapper/320:20:293
1057799210,3cyclictest18455-21fschecks_time00:30:193
1057799210,1cyclictest0-21swapper/321:10:323
1057799210,19cyclictest0-21swapper/300:25:143
1057799210,0cyclictest0-21swapper/320:50:133
10575992119,1cyclictest0-21swapper/200:35:132
10575992118,2cyclictest0-21swapper/223:55:132
1057599210,20cyclictest0-21swapper/223:20:392
1057599210,1cyclictest0-21swapper/219:15:122
1057599210,19cyclictest0-21swapper/223:06:322
1057799200,1cyclictest0-21swapper/323:53:393
1057799200,18cyclictest0-21swapper/300:00:333
1057799200,17cyclictest0-21swapper/320:05:143
1057599200,20cyclictest0-21swapper/222:08:452
1057599200,20cyclictest0-21swapper/222:08:442
1057099200,7cyclictest0-21swapper/122:44:131
1057799195,13cyclictest24793-21nfs4_client00:35:213
1057799190,2cyclictest0-21swapper/320:45:273
1057799190,18cyclictest0-21swapper/320:35:113
10575991915,3cyclictest32-21ksoftirqd/220:40:012
1057599190,19cyclictest0-21swapper/222:53:372
10570991919,0cyclictest0-21swapper/122:54:081
1057799183,12cyclictest0-21swapper/319:40:583
1057799181,2cyclictest16782-21aten2_r7power_e20:30:153
1057799180,1cyclictest0-21swapper/320:15:143
1057799180,17cyclictest19263-21uname19:25:243
1057799180,15cyclictest0-21swapper/320:00:243
1057599180,17cyclictest0-21swapper/222:58:102
1057599180,17cyclictest0-21swapper/200:27:542
1057099180,1cyclictest0-21swapper/122:12:001
1057099180,1cyclictest0-21swapper/121:18:581
10567991817,1cyclictest0-21swapper/000:37:090
1057799170,3cyclictest0-21swapper/319:55:113
1057599174,7cyclictest224-21systemd-journal23:05:002
1057599170,15cyclictest0-21swapper/222:11:182
1057599170,0cyclictest0-21swapper/200:17:362
10570991710,2cyclictest23802-21cron20:45:001
1057099170,17cyclictest0-21swapper/122:58:511
1056799176,5cyclictest17267-21munin-run00:29:590
10567991717,0cyclictest0-21swapper/000:02:130
1056799171,1cyclictest141rcu_preempt21:35:380
1057799168,7cyclictest29183-21sort19:45:313
1057799161,1cyclictest0-21swapper/321:00:163
1057799160,15cyclictest0-21swapper/319:25:003
1057599160,16cyclictest0-21swapper/221:18:262
1057099164,5cyclictest3290-21diskmemload22:30:511
10570991613,2cyclictest0-21swapper/121:45:241
1056799160,2cyclictest0-21swapper/022:34:180
1056799160,1cyclictest0-21swapper/000:10:120
166822150,1sleep20-21swapper/221:42:062
1057799158,6cyclictest12706-21chrt19:12:453
1057599153,2cyclictest224-21systemd-journal23:36:592
10575991513,1cyclictest0-21swapper/223:26:292
10575991512,3cyclictest0-21swapper/222:30:572
1057599150,15cyclictest0-21swapper/221:26:052
1057599150,14cyclictest0-21swapper/223:42:042
1057599150,13cyclictest0-21swapper/200:01:032
1056799157,7cyclictest7804-21munin-run23:10:000
10567991510,4cyclictest1291-21runrttasks22:56:100
1056799150,6cyclictest224-21systemd-journal00:35:000
10575991413,1cyclictest0-21swapper/222:39:352
10575991410,3cyclictest23224-21wc20:40:352
1057599140,13cyclictest0-21swapper/223:46:332
1057599134,6cyclictest311rcuc/222:45:002
10575991310,2cyclictest8889-21grep20:10:292
10575991310,2cyclictest8889-21grep20:10:282
10575991310,2cyclictest7751-21turbostat22:00:002
10575991310,2cyclictest707-21grep19:55:172
10575991310,2cyclictest5369-21grep20:05:162
10575991310,2cyclictest2611-21tr20:00:162
10575991310,2cyclictest20390-21grep00:05:172
10575991310,2cyclictest17185-21cron22:30:002
1057099139,3cyclictest7232-21grep22:45:221
1057099139,3cyclictest28426-21sed00:10:291
1057099131,11cyclictest1222-21apache_volume23:05:111
10570991310,2cyclictest26829-21wc23:45:281
10570991310,2cyclictest24212-21grep20:45:131
10570991310,2cyclictest2092-21cron19:59:591
1057099130,11cyclictest30828-21cut19:50:201
1056799136,6cyclictest10032-21munin-run22:25:010
1056799134,4cyclictest8323-21aten2_r7power_p22:00:130
1056799134,4cyclictest8323-21aten2_r7power_p22:00:130
10567991312,1cyclictest0-21swapper/023:37:510
10567991310,2cyclictest6518-21sed20:05:320
10567991310,2cyclictest31071-21tr22:40:010
10567991310,2cyclictest29204-21grep22:15:150
10567991310,2cyclictest14949-21grep21:40:260
1056799130,0cyclictest0-21swapper/019:25:140
1057599129,2cyclictest6571-21ssh22:45:182
1057599129,2cyclictest32360-21seq21:00:242
1057599129,2cyclictest30173-21cut22:15:212
1057599129,2cyclictest25536-21tr19:40:142
1057599128,3cyclictest5240-21apache221:35:252
1057599128,3cyclictest26948-21tr00:10:142
1057599127,4cyclictest10637-21wc20:15:232
1057599126,3cyclictest224-21systemd-journal19:26:492
1057599124,4cyclictest8443-21ssh22:00:152
1057599124,4cyclictest8443-21ssh22:00:142
1057599124,4cyclictest224-21systemd-journal20:45:012
1057599120,11cyclictest6374-21tr00:20:112
1057099129,2cyclictest908-21turbostat21:05:001
1057099129,2cyclictest3083-21sleep22:20:001
1057099129,2cyclictest27507-21grep23:00:201
1057099129,2cyclictest25690-21grep22:35:191
1057099129,2cyclictest25040-21tr19:39:591
1057099129,2cyclictest20867-21head00:05:191
1057099129,2cyclictest17694-21cron23:39:591
1057099129,2cyclictest16473-21cut22:05:221
1057099129,2cyclictest16473-21cut22:05:211
1057099129,2cyclictest16163-21grep19:20:171
1057099129,2cyclictest13077-21ls00:25:221
1057099129,2cyclictest12080-21tr20:20:121
1057099129,2cyclictest11614-21wc00:25:001
1057099128,3cyclictest21475-21sort20:40:011
1057099128,2cyclictest7706-21grep21:35:271
1057099128,2cyclictest14454-21grep00:00:171
1057099122,1cyclictest26459-21apt-get20:50:131
1057099120,1cyclictest0-21swapper/120:30:131
1057099120,11cyclictest13891-21grep19:15:171
1057099120,10cyclictest0-21swapper/119:45:131
1056799129,2cyclictest928-21cat21:55:000
1056799129,2cyclictest7896-21grep00:20:280
1056799129,2cyclictest6839-21tail22:45:190
1056799129,2cyclictest5703-21wc00:20:000
1056799129,2cyclictest5700-21tail21:10:310
1056799129,2cyclictest5015-21chrt22:44:340
1056799129,2cyclictest32141-21cron19:55:000
1056799129,2cyclictest27932-21grep20:50:300
1056799129,2cyclictest25326-21tr19:40:130
1056799129,2cyclictest20670-21df00:05:160
1056799129,2cyclictest16606-21head23:15:220
1056799129,2cyclictest14649-21cat19:15:290
1056799129,2cyclictest10594-21cron23:35:000
1056799128,3cyclictest11037-21tr22:25:150
1056799128,1cyclictest0-21swapper/021:48:060
1056799127,4cyclictest751-21latency_hist23:05:000
1056799125,2cyclictest0-21swapper/020:45:120
10567991211,1cyclictest0-21swapper/023:58:040
1056799120,1cyclictest0-21swapper/021:55:490
1056799120,11cyclictest30217-21sed23:25:180
1057599118,2cyclictest28931-21cut20:55:142
1057599118,2cyclictest28840-21sed21:50:312
1057599118,2cyclictest18859-21ssh21:20:202
1057599118,2cyclictest18240-21sh00:30:172
1057599118,2cyclictest16415-21tail23:15:182
1057599118,2cyclictest13617-21tail20:20:282
1057599117,3cyclictest31717-21grep21:30:122
1057599117,3cyclictest23856-21wc19:35:192
1057599117,3cyclictest12010-21tr19:10:242
1057599116,4cyclictest15610-21wc20:25:242
1057599116,3cyclictest4274-21grep23:30:152
1057599113,4cyclictest224-21systemd-journal20:51:272
1057599112,6cyclictest224-21systemd-journal19:46:492
1057599110,3cyclictest0-21swapper/222:24:052
1057099119,1cyclictest29725-21sh23:25:131
1057099118,2cyclictest6633-21ssh23:55:131
1057099118,2cyclictest6013-21ssh21:10:421
1057099118,2cyclictest3809-21tr22:20:151
1057099118,2cyclictest350-21tr00:15:141
1057099118,2cyclictest28486-21ssh00:39:211
1057099118,2cyclictest27807-21cut21:50:191
1057099118,2cyclictest22100-21ssh23:20:031
1057099118,2cyclictest21301-21grep19:30:241
1057099118,2cyclictest19061-21cut21:20:211
1057099118,2cyclictest18996-21tail23:40:161
1057099118,2cyclictest18532-21ssh00:30:211
1057099118,2cyclictest18060-21cat19:25:001
1057099118,2cyclictest17847-21ssh23:16:081
1057099118,2cyclictest15288-21tr20:25:201
1057099118,2cyclictest1527-21grep21:55:141
1057099118,2cyclictest14738-21sed22:05:011
1057099118,2cyclictest14738-21sed22:05:011
1057099118,2cyclictest11219-21sed20:15:311
1057099114,6cyclictest12018-21ssh23:12:201
1057099110,4cyclictest0-21swapper/122:25:191
1056799119,1cyclictest28309-21grep19:45:180
1056799118,2cyclictest9178-21cat23:10:180
1056799118,2cyclictest7178-21basename20:10:100
1056799118,2cyclictest7178-21basename20:10:100
1056799118,2cyclictest3442-21ls21:10:000
1056799118,2cyclictest31771-21df21:00:170
1056799118,2cyclictest25638-21df23:45:150
1056799118,2cyclictest25046-21basename19:40:000
1056799118,2cyclictest21947-21tr20:40:110
1056799118,2cyclictest21196-21cron20:40:010
1056799118,2cyclictest20117-21grep23:40:290
1056799118,2cyclictest18149-21ssh22:53:550
1056799118,2cyclictest1263-21ssh21:30:350
1056799118,2cyclictest11324-21chrt20:15:320
1056799116,1cyclictest24263-21grep23:20:320
1057599108,1cyclictest1124-21chrt21:05:022
1057599107,2cyclictest21237-21grep19:30:242
1057599107,2cyclictest15809-21grep19:20:152
1057599101,5cyclictest224-21systemd-journal19:50:012
1057599101,3cyclictest224-21systemd-journal20:30:012
1057099108,1cyclictest9066-21seq20:12:081
1057099108,1cyclictest9066-21seq20:12:071
1057099108,1cyclictest3911-21ssh23:30:091
1057099108,1cyclictest3172-21ssh23:52:051
1057099108,1cyclictest29041-21ls20:55:151
1057099108,1cyclictest2340-21date20:00:011
1057099108,1cyclictest11937-21ntpq19:10:231
1057099107,2cyclictest31424-21ssh21:30:041
1057099107,2cyclictest26057-21head19:40:171
1057099106,3cyclictest2437-21ps21:05:251
1057099100,9cyclictest12943-21rm21:40:021
1056799108,2cyclictest4616-21cat20:05:010
1056799108,1cyclictest21280-21rm21:22:020
1056799108,1cyclictest16177-21cut19:20:170
1056799108,1cyclictest14566-21wc20:25:140
1056799107,2cyclictest31808-21ssh23:50:050
1056799107,2cyclictest24452-21taskset21:25:020
1056799107,2cyclictest17403-21ssh22:05:440
1056799107,2cyclictest17403-21ssh22:05:440
1056799107,2cyclictest1367-21cut19:55:260
1056799103,3cyclictest224-21systemd-journal21:15:000
105709990,5cyclictest224-21systemd-journal20:06:061
105679997,1cyclictest29660-21cat20:55:190
105679997,1cyclictest20518-21apache_accesses19:30:100
105679997,1cyclictest10846-21basename19:10:150
105679996,2cyclictest17713-21cat20:30:210
105679990,6cyclictest14092-21basename20:25:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional