You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-25 - 10:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 25, 2024 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2025729986,9sleep10-21swapper/119:06:031
2023229885,9sleep30-21swapper/319:05:443
2044528464,15sleep20-21swapper/219:08:292
2052128270,8sleep00-21swapper/019:09:270
24522720,0sleep10-21swapper/121:28:111
47672280,0chrt4768-21sh00:09:023
2066399280,28cyclictest0-21swapper/223:13:292
20661992522,2cyclictest0-21swapper/122:10:001
2066699240,18cyclictest0-21swapper/322:50:033
20663992321,1cyclictest9880-21apt-get22:40:142
2066399230,19cyclictest0-21swapper/223:15:032
2066699220,3cyclictest27502-21cron22:30:003
20666992121,0cyclictest0-21swapper/322:14:503
20663992019,1cyclictest0-21swapper/223:08:052
20663992013,6cyclictest8624-21diskmemload22:25:462
20663992012,7cyclictest19341-21sh20:25:002
2065699206,6cyclictest29301-21munin-run19:35:000
2066699192,16cyclictest0-21swapper/322:20:103
2066399192,4cyclictest32-21ksoftirqd/221:17:062
20663991917,1cyclictest0-21swapper/222:46:532
20661991916,2cyclictest5240-21apache200:18:531
2065699190,19cyclictest0-21swapper/021:59:180
237422180,0sleep00-21swapper/022:26:200
2066399189,8cyclictest0-21swapper/223:50:142
20663991811,6cyclictest8624-21diskmemload21:40:522
20663991810,7cyclictest8624-21diskmemload22:20:402
20661991818,0cyclictest0-21swapper/121:16:331
2066199180,1cyclictest0-21swapper/121:46:531
2066399176,11cyclictest8624-21diskmemload21:51:062
2066399172,13cyclictest0-21swapper/221:59:362
20663991715,2cyclictest311rcuc/221:48:572
20663991715,1cyclictest8624-21diskmemload23:39:312
20663991715,1cyclictest472-21avahi-daemon00:00:022
2066399171,15cyclictest0-21swapper/222:12:332
2066199170,14cyclictest0-21swapper/123:05:171
2065699175,1cyclictest0-21swapper/021:45:130
20666991610,5cyclictest8624-21diskmemload21:53:063
2066699160,14cyclictest0-21swapper/323:23:533
2066399167,8cyclictest8624-21diskmemload00:12:122
2066399166,7cyclictest8624-21diskmemload21:10:302
2066399161,2cyclictest23029-21expr00:25:232
20663991610,5cyclictest8624-21diskmemload21:37:072
2066399160,1cyclictest0-21swapper/221:09:352
2065699164,6cyclictest24081-21perf22:50:000
2065699160,1cyclictest0-21swapper/020:05:140
2066699150,1cyclictest0-21swapper/300:29:323
2066399158,6cyclictest21656-21sh23:34:232
2066399157,8cyclictest0-21swapper/223:00:132
2066399153,7cyclictest31000-21munin-run22:55:002
2066399153,10cyclictest30366-21cron22:10:002
2066399152,12cyclictest8624-21diskmemload23:47:402
20663991513,2cyclictest0-21swapper/221:25:292
2066399150,14cyclictest0-21swapper/222:18:072
2066199152,8cyclictest8624-21diskmemload22:50:281
20661991510,2cyclictest25-21ksoftirqd/100:09:241
2066199150,15cyclictest0-21swapper/123:55:451
2065699154,10cyclictest15884-21/usr/sbin/munin22:00:240
20656991515,0cyclictest0-21swapper/021:10:210
2065699150,1cyclictest0-21swapper/022:37:440
2065699150,14cyclictest0-21swapper/021:19:420
2066699148,5cyclictest20651-21cyclictest22:37:053
2066699142,4cyclictest0-21swapper/319:30:103
20666991411,2cyclictest724-21taskset00:34:533
20666991410,3cyclictest2483-21awk19:50:003
2066399149,3cyclictest8624-21diskmemload22:35:392
2066399147,7cyclictest0-21swapper/219:31:472
2066399146,8cyclictest0-21swapper/200:34:082
20663991414,0cyclictest0-21swapper/223:29:202
20663991412,1cyclictest0-21swapper/219:20:092
2066399141,12cyclictest28095-21wc20:40:202
2066399141,12cyclictest15331-21sed20:15:182
2066399140,2cyclictest32-21ksoftirqd/222:04:342
2066399140,2cyclictest0-21swapper/222:57:402
2066399140,12cyclictest32-21ksoftirqd/223:55:122
2065699141,6cyclictest29632-21munin-run21:25:000
2065699140,1cyclictest141rcu_preempt00:29:570
2066699139,3cyclictest801-21cat21:50:013
2066699139,3cyclictest7914-21tail22:15:283
2066699139,3cyclictest27618-21grep23:15:193
2066699139,3cyclictest19453-21taskset21:16:403
2066699139,3cyclictest18408-21ssh22:00:533
20666991311,2cyclictest0-21swapper/323:43:433
20666991310,2cyclictest18025-21head20:20:193
20666991310,2cyclictest15773-21cron21:15:003
2066399139,3cyclictest32-21ksoftirqd/220:25:152
2066399139,2cyclictest8681-21wc20:00:262
2066399137,6cyclictest0-21swapper/221:31:142
2066399137,5cyclictest0-21swapper/200:35:262
2066399131,8cyclictest15576-21sh00:17:522
20663991311,1cyclictest8624-21diskmemload23:24:292
20663991310,2cyclictest3424-21grep20:55:302
20663991310,2cyclictest32235-21cut00:05:132
20663991310,2cyclictest26626-21sort20:40:002
2066399130,11cyclictest32-21ksoftirqd/220:45:142
2066199139,3cyclictest6313-21wc21:05:001
2066199139,3cyclictest3176-21tr19:50:171
2066199139,3cyclictest27046-21sh00:30:001
2066199139,2cyclictest15563-21cat23:30:011
2066199135,5cyclictest0-21swapper/123:15:001
20661991310,2cyclictest7597-21wc23:00:291
20661991310,2cyclictest7111-21cut21:05:191
20661991310,2cyclictest30936-21cut22:10:141
20661991310,2cyclictest25-21ksoftirqd/119:20:001
20661991310,2cyclictest2488-21cut23:20:171
20661991310,2cyclictest16257-21cut22:00:141
2066199130,1cyclictest0-21swapper/121:57:141
2066199130,12cyclictest23294-21grep23:35:141
2066199130,12cyclictest2049-21head00:35:211
2066199130,11cyclictest5083-21grep21:30:171
2065699139,3cyclictest24595-21tr20:35:150
2065699139,3cyclictest19722-21tr23:10:110
2065699139,3cyclictest12716-21cut20:10:150
2065699138,4cyclictest308-21ssh23:41:340
20656991310,2cyclictest6843-21head22:15:160
20656991310,2cyclictest6696-21sed21:30:330
20656991310,2cyclictest4618-21sed19:50:330
20656991310,2cyclictest31921-21grep20:50:130
20656991310,2cyclictest31012-21sort21:25:190
20656991310,2cyclictest14446-21ssh23:05:250
20656991310,2cyclictest13356-21grep00:15:260
20656991310,2cyclictest10654-21head22:40:200
2065699130,12cyclictest24220-21grep22:05:180
2065699130,12cyclictest20460-21cat22:25:000
2066699129,2cyclictest9503-21grep23:25:163
2066699129,2cyclictest7987-21grep20:00:183
2066699129,2cyclictest30186-21head20:45:183
2066699129,2cyclictest2315-21wc22:35:003
2066699129,2cyclictest1472-21ssh00:35:173
2066699129,2cyclictest10855-21sed20:05:213
2066699128,3cyclictest25470-21grep20:35:223
2066699128,3cyclictest23457-21grep21:20:123
2066699128,2cyclictest13068-21sort20:10:183
2066699128,1cyclictest8624-21diskmemload23:45:293
2066699124,5cyclictest39-21ksoftirqd/321:30:153
20666991211,1cyclictest0-21swapper/323:34:193
2066699120,11cyclictest32313-21awk19:45:003
2066399129,2cyclictest31411-21sed19:40:212
2066399129,2cyclictest12758-21cut20:10:152
2066399128,3cyclictest956-21ssh23:42:112
2066399128,3cyclictest4598-21grep19:50:332
2066399128,3cyclictest11797-21head20:05:332
2066399128,2cyclictest26961-21cat19:30:002
2066399128,2cyclictest26961-21cat19:30:002
2066399128,2cyclictest21876-21sort19:10:202
2066399125,7cyclictest0-21swapper/219:55:172
2066399124,4cyclictest224-21systemd-journal22:30:142
2066199129,2cyclictest9361-21cron22:40:001
2066199129,2cyclictest27784-21grep20:40:161
2066199129,2cyclictest26498-21grep00:00:191
2066199129,2cyclictest22009-21awk19:10:211
2066199129,2cyclictest20943-21grep21:40:321
2066199128,3cyclictest31315-21tr19:40:191
2066199126,5cyclictest8624-21diskmemload22:59:401
2066199122,5cyclictest224-21systemd-journal20:25:001
20661991212,0cyclictest0-21swapper/100:33:291
2066199120,12cyclictest0-21swapper/120:34:461
2066199120,11cyclictest12658-21chrt21:12:041
2065699129,2cyclictest9485-21ssh23:48:420
2065699129,2cyclictest8117-21sed00:10:340
2065699129,2cyclictest7602-21tail21:05:250
2065699129,2cyclictest28143-21head20:40:200
2065699129,2cyclictest27788-21grep00:00:320
2065699129,2cyclictest2487-21grep23:20:170
2065699129,2cyclictest24696-21sed23:35:290
2065699129,2cyclictest24650-21grep22:50:140
2065699129,2cyclictest17971-21sed00:20:190
2065699129,2cyclictest13-21ksoftirqd/022:56:530
2065699129,2cyclictest1026-21grep19:45:170
280672110,2sleep10-21swapper/122:30:151
2066699118,2cyclictest9977-21grep22:40:163
2066699118,2cyclictest8570-21ssh23:01:113
2066699118,2cyclictest7348-21taskset19:59:093
2066699118,2cyclictest6479-21ssh00:10:173
2066699118,2cyclictest3062-21ssh22:57:363
2066699118,2cyclictest30617-21tr21:25:153
2066699118,2cyclictest29998-21cat19:40:003
2066699118,2cyclictest29281-21ssh00:02:163
2066699118,2cyclictest27472-21ssh23:38:123
2066699118,2cyclictest25146-21cat19:20:173
2066699118,2cyclictest23864-21cat19:20:003
2066699118,2cyclictest20911-21ssh23:55:563
2066699118,2cyclictest19922-21grep23:10:123
2066699118,2cyclictest19620-21ssh21:40:203
2066699118,2cyclictest17518-21sh23:54:343
2066699118,2cyclictest16708-21cat22:45:003
2066699118,2cyclictest16504-21grep20:15:363
2066699118,2cyclictest15865-21ssh23:06:323
2066699118,2cyclictest15785-21ssh21:37:243
2066699118,2cyclictest14054-21taskset21:58:413
2066699118,2cyclictest12527-21expr00:15:183
2066699116,2cyclictest3376-21grep20:55:303
2066699115,5cyclictest20886-21apt-get19:10:113
2066699114,2cyclictest224-21systemd-journal22:06:103
2066699110,10cyclictest4548-21date19:50:323
2066699110,10cyclictest18161-21tr00:20:223
2066399118,2cyclictest455-21grep20:50:212
2066399118,2cyclictest23458-21tr21:20:122
2066399115,3cyclictest311rcuc/200:24:402
2066199118,2cyclictest8936-21ssh00:11:331
2066199118,2cyclictest5890-21taskset21:53:011
2066199118,2cyclictest4268-21ssh23:45:121
2066199118,2cyclictest32699-21seq23:41:301
2066199118,2cyclictest30040-21wc19:40:001
2066199118,2cyclictest29634-21grep20:45:151
2066199118,2cyclictest26815-21ssh23:15:091
2066199118,2cyclictest25181-21idleruntime20:35:191
2066199118,2cyclictest23174-21ssh22:25:461
2066199118,2cyclictest17740-21ssh22:45:151
2066199118,2cyclictest17698-21grep00:20:171
2066199118,2cyclictest15850-21sh23:30:131
2066199118,2cyclictest15523-21ssh21:37:101
2066199118,2cyclictest12994-21ssh22:19:561
2066199118,2cyclictest11795-21grep20:05:331
2066199118,2cyclictest10031-21tr22:40:161
2066199117,3cyclictest24416-21uname21:20:191
2066199117,3cyclictest14412-21head22:20:161
2066199117,3cyclictest12685-21head23:50:241
2065699118,2cyclictest32399-21wc00:05:150
2065699118,2cyclictest32025-21ssh22:32:280
2065699118,2cyclictest30793-21sort19:40:150
2065699118,2cyclictest2965-21head20:55:250
2065699118,2cyclictest27796-21cat00:30:160
2065699118,2cyclictest24850-21head19:20:140
2065699118,2cyclictest21101-21ssh21:40:410
2065699118,2cyclictest20070-21ssh23:55:280
2065699118,2cyclictest18184-21ssh21:39:450
2065699118,2cyclictest18010-21tail20:20:190
2065699118,2cyclictest1408-21grep00:35:160
2065699118,2cyclictest12420-21cron23:05:000
2065699118,2cyclictest11166-21ssh23:50:060
2065699117,3cyclictest28855-21ssh23:15:320
2065699113,4cyclictest224-21systemd-journal20:17:000
2065699111,8cyclictest31091-21cpuspeed_turbos22:10:150
2065699111,6cyclictest5314-21apache_accesses19:55:110
2065699110,10cyclictest23514-21sshd19:18:470
2065699110,10cyclictest20357-21grep20:25:180
2065699110,10cyclictest18906-21ssh23:31:400
2066699108,1cyclictest7843-21which21:05:283
2066699108,1cyclictest32213-21cat20:50:143
2066699108,1cyclictest23835-21taskset20:31:463
2066699107,2cyclictest5312-21awk21:00:253
2066699107,2cyclictest19612-21apt20:25:123
2066399108,2cyclictest22530-21sed20:30:212
2066399108,2cyclictest0-21swapper/219:45:122
2066399107,2cyclictest29651-21chrt19:36:532
2066399107,1cyclictest311rcuc/219:17:412
2066399106,3cyclictest0-21swapper/221:00:122
2066199108,1cyclictest3736-21taskset20:58:451
2066199108,1cyclictest27714-21df_abs19:30:131
2066199108,1cyclictest2119-21taskset19:45:411
2066199107,2cyclictest7955-21expr20:00:181
2066199107,2cyclictest26511-21taskset19:25:471
2066199107,2cyclictest26511-21taskset19:25:471
2066199107,2cyclictest24876-21head19:20:141
2066199107,2cyclictest20233-21grep20:25:161
2066199107,2cyclictest15071-21tr20:15:161
2066199107,2cyclictest12428-21chrt20:10:061
2066199106,3cyclictest7421-21cron20:00:001
2066199104,2cyclictest224-21systemd-journal20:50:001
2065699108,1cyclictest8725-21ssh23:25:090
2065699108,1cyclictest8261-21df_abs20:00:190
2065699108,1cyclictest29879-21cat20:45:160
2065699108,1cyclictest297892sleep019:38:390
2065699108,1cyclictest23161-21ntp_states20:30:300
2065699108,1cyclictest2203-21sed21:50:200
2065699107,2cyclictest4139-21dpkg21:00:130
2065699107,2cyclictest26499-21chrt19:25:370
2065699107,2cyclictest26499-21chrt19:25:370
2065699102,4cyclictest22729-21sshd19:11:470
206669997,1cyclictest26620-21taskset19:27:093
206669997,1cyclictest26620-21taskset19:27:093
206669996,2cyclictest28176-21grep20:40:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional