You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-17 - 00:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Jun 16, 2024 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21872210375,9sleep30-21swapper/319:06:533
21872210375,9sleep30-21swapper/319:06:533
2193229862,31sleep10-21swapper/119:07:401
2193229862,31sleep10-21swapper/119:07:401
2186628667,15sleep20-21swapper/219:06:482
2186628667,15sleep20-21swapper/219:06:482
2183228166,10sleep00-21swapper/019:06:220
2183228166,10sleep00-21swapper/019:06:220
2220599240,22cyclictest0-21swapper/121:56:001
2220599230,20cyclictest0-21swapper/121:20:191
2220399238,8cyclictest17092-21munin-run21:55:000
22209992116,4cyclictest0-21swapper/222:00:292
2221399200,1cyclictest0-21swapper/322:17:353
2221399200,19cyclictest0-21swapper/322:33:433
2220999200,1cyclictest0-21swapper/222:26:322
2220399190,0cyclictest0-21swapper/021:10:450
2221399181,16cyclictest0-21swapper/323:51:373
2220999181,2cyclictest141rcu_preempt23:01:392
2220999180,1cyclictest5987-21aten2_r7power_v20:50:132
2220999180,18cyclictest0-21swapper/223:47:232
2220399184,6cyclictest16060-21perf20:05:000
2220399180,1cyclictest141rcu_preempt22:16:570
2220399180,0cyclictest0-21swapper/000:10:480
22209991715,2cyclictest0-21swapper/223:10:122
2220999170,9cyclictest0-21swapper/223:25:012
2220999170,17cyclictest0-21swapper/223:32:332
2220599171,3cyclictest141rcu_preempt00:01:351
2220999164,8cyclictest26101-21runrttasks22:24:162
22209991611,1cyclictest32-21ksoftirqd/221:38:402
2220599165,10cyclictest21531-21gdbus00:07:231
2220399163,6cyclictest21511-21apache223:05:000
2220399160,2cyclictest0-21swapper/000:30:500
2220399160,1cyclictest0-21swapper/019:55:120
2220399160,15cyclictest21302-21apache_processe20:15:090
2220999155,6cyclictest224-21systemd-journal20:06:422
2220999155,5cyclictest224-21systemd-journal21:10:002
22209991511,3cyclictest28378-21grep23:35:382
2220999150,7cyclictest224-21systemd-journal22:12:152
22205991510,3cyclictest14443-21cron00:14:591
2220399157,8cyclictest1287-21minissdpd22:23:100
22203991513,1cyclictest0-21swapper/023:07:510
2220399150,1cyclictest0-21swapper/000:29:150
2221399148,2cyclictest22373-21apt-get23:55:133
2221399147,3cyclictest12877-21ssh21:30:153
22213991414,0cyclictest0-21swapper/323:01:043
22213991413,1cyclictest0-21swapper/322:07:253
22213991413,1cyclictest0-21swapper/321:00:133
22213991413,1cyclictest0-21swapper/321:00:133
22213991410,3cyclictest9861-21grep00:10:193
22213991410,2cyclictest24731-21cat19:15:013
2221399140,2cyclictest0-21swapper/320:25:123
2221399140,1cyclictest0-21swapper/321:29:283
2221399140,12cyclictest30231-21cat21:20:013
2220999149,4cyclictest17162-21wc21:55:002
22209991411,2cyclictest30583-21wc19:25:252
22209991410,3cyclictest1341-21tr20:40:162
22209991410,3cyclictest10711-21grep19:50:332
2220599149,3cyclictest391-21cron00:30:001
22205991411,1cyclictest0-21swapper/121:30:121
22205991410,3cyclictest23805-21sed23:55:271
22205991410,2cyclictest24398-21cut20:20:201
2220599140,13cyclictest3812-21basename22:55:141
2220399146,7cyclictest12076-21idleruntime-cro23:25:010
22203991410,4cyclictest0-21swapper/023:13:150
22203991410,3cyclictest31220-21sed19:25:360
22203991410,2cyclictest8065-21chrt00:35:240
22203991410,2cyclictest8065-21chrt00:35:240
2220399140,1cyclictest141rcu_preempt21:27:480
2220399140,1cyclictest0-21swapper/021:22:050
2221399139,3cyclictest1612-21cut00:30:173
2221399139,2cyclictest22403-21cat20:15:233
2221399135,7cyclictest11728-21gdbus00:07:233
22213991312,1cyclictest39-21ksoftirqd/323:09:153
22213991310,2cyclictest29927-21tr20:30:313
22213991310,2cyclictest23098-21cut22:45:153
2221399130,13cyclictest0-21swapper/321:43:373
2221399130,12cyclictest20642-21tr22:20:163
2220999132,7cyclictest2962-21sh22:30:262
22209991310,2cyclictest30333-21wc20:30:382
22209991310,2cyclictest2691-21grep00:30:322
22209991310,2cyclictest23672-21head19:10:262
22209991310,2cyclictest17884-21basename21:55:162
22209991310,2cyclictest11313-21cut21:00:192
22209991310,2cyclictest11313-21cut21:00:182
2220999130,1cyclictest0-21swapper/222:08:562
2220599139,3cyclictest31221-21tr20:35:141
2220599139,3cyclictest13731-21wc20:00:001
2220599139,2cyclictest31260-21cat23:15:011
2220599135,5cyclictest16438-21wc23:50:261
2220599134,5cyclictest15341-21diskmemload00:39:111
2220599134,5cyclictest15341-21diskmemload00:39:111
2220599132,7cyclictest141rcu_preempt23:35:171
22205991311,1cyclictest0-21swapper/122:22:241
22205991310,2cyclictest8304-21grep19:45:351
22205991310,2cyclictest6598-21grep20:50:201
22205991310,2cyclictest21184-21hostname00:20:171
22205991310,2cyclictest16692-21grep21:10:201
2220599130,11cyclictest0-21swapper/119:40:131
2220399134,4cyclictest493-21systemd-logind00:07:230
2220399131,6cyclictest8003-21munin-run22:35:000
22203991310,2cyclictest9718-21perf21:49:590
22203991310,2cyclictest31732-21cat22:05:000
22203991310,2cyclictest30953-21grep20:35:000
22203991310,2cyclictest26505-21grep19:15:340
22203991310,2cyclictest24552-21munin-run21:59:590
2220399130,8cyclictest24559-21idleruntime-cro19:15:000
2221399129,2cyclictest7723-21cat22:10:203
2221399129,2cyclictest6977-21wc19:45:173
2221399129,2cyclictest4574-21ssh22:55:203
2221399129,2cyclictest3491-21grep21:45:213
2221399129,2cyclictest29838-21tr19:25:183
2221399129,2cyclictest28337-21wc00:25:253
2221399129,2cyclictest25928-21turbostat.cron20:25:003
2221399129,2cyclictest25409-21cut22:00:183
2221399129,2cyclictest19741-21chrt21:12:153
2221399129,2cyclictest10503-21grep21:50:163
2221399129,2cyclictest10075-21grep23:45:373
22213991211,1cyclictest0-21swapper/300:19:413
2220999129,2cyclictest877-21wc20:40:002
2220999129,2cyclictest6939-21tr19:45:172
2220999129,2cyclictest4039-21systemctl19:40:012
2220999129,2cyclictest32-21ksoftirqd/221:24:072
2220999129,2cyclictest27842-21tail20:25:322
2220999129,2cyclictest25088-21grep19:15:152
2220999129,2cyclictest24815-21sed22:45:352
2220999129,2cyclictest15698-21tr22:40:002
2220999129,2cyclictest14347-21grep22:15:202
2220999129,2cyclictest11633-21tr19:55:142
2220999128,3cyclictest7599-21taskset00:08:092
2220999128,2cyclictest19642-21tail23:05:352
2220999126,2cyclictest224-21systemd-journal21:20:002
2220999121,6cyclictest493-21systemd-logind21:45:302
22209991211,1cyclictest32-21ksoftirqd/222:40:082
2220999121,10cyclictest0-21swapper/219:40:132
2220999120,12cyclictest0-21swapper/200:26:452
2220999120,0cyclictest0-21swapper/200:37:172
2220999120,0cyclictest0-21swapper/200:37:162
2220599129,3cyclictest25-21ksoftirqd/123:21:201
2220599129,2cyclictest5513-21grep23:20:011
2220599129,2cyclictest32665-21grep19:30:161
2220599129,2cyclictest31533-21sed22:50:361
2220599129,2cyclictest31533-21sed22:50:361
2220599129,2cyclictest30341-21sed19:25:221
2220599129,2cyclictest28721-21grep22:25:271
2220599129,2cyclictest23059-21grep22:45:151
2220599129,2cyclictest19389-21grep20:10:221
2220599129,2cyclictest14466-21grep21:05:291
2220599129,2cyclictest14374-21tail22:15:201
2220599129,2cyclictest13024-21cp21:05:011
2220599129,2cyclictest12728-21grep23:00:361
2220599129,2cyclictest10247-21rt-features22:35:301
2220599128,3cyclictest16041-21dump-pmu-power20:05:001
2220599128,3cyclictest10306-21ssh21:50:161
2220599128,2cyclictest18401-21cron20:10:001
2220399129,3cyclictest21512-21apache222:44:430
2220399129,2cyclictest7939-21ls22:10:240
2220399129,2cyclictest7556-21grep20:50:340
2220399129,2cyclictest30884-21grep22:50:280
2220399129,2cyclictest30884-21grep22:50:280
2220399129,2cyclictest27719-21ntpq23:35:290
2220399129,2cyclictest27471-21cut19:20:140
2220399129,2cyclictest24983-21cat21:15:310
2220399129,2cyclictest20206-21grep23:30:190
2220399129,2cyclictest17638-21wc20:05:350
2220399129,2cyclictest13771-21head21:30:210
2220399129,2cyclictest12360-21head21:00:350
2220399129,2cyclictest12360-21head21:00:350
2220399128,3cyclictest6158-21latency_hist22:10:000
2220399128,2cyclictest28529-21ssh21:40:220
2220399125,6cyclictest17813-21ssh00:16:430
22203991210,1cyclictest19410-21head20:10:220
22203991210,1cyclictest13857-21cut21:05:210
22203991210,1cyclictest13785-21strings23:25:250
2221399119,1cyclictest27981-21grep19:20:183
2221399118,2cyclictest9888-21ssh22:35:263
2221399118,2cyclictest9111-21ssh00:35:403
2221399118,2cyclictest9111-21ssh00:35:403
2221399118,2cyclictest524-21taskset22:51:583
2221399118,2cyclictest524-21taskset22:51:583
2221399118,2cyclictest5209-21grep20:45:333
2221399118,2cyclictest32214-21ssh23:39:143
2221399118,2cyclictest32135-21sed23:15:183
2221399118,2cyclictest31446-21grep21:20:223
2221399118,2cyclictest31408-21grep20:35:153
2221399118,2cyclictest28025-21cut22:25:203
2221399118,2cyclictest25982-21ntpq23:10:273
2221399118,2cyclictest25679-21ssh00:23:413
2221399118,2cyclictest22823-21sh21:58:183
2221399118,2cyclictest20471-21tr21:35:163
2221399118,2cyclictest1670-21df20:40:183
2221399118,2cyclictest16171-21basename22:40:133
2221399118,2cyclictest15083-21taskset23:26:073
2221399118,2cyclictest1414-21tr23:40:163
2221399118,2cyclictest11008-21taskset19:53:143
2221399116,4cyclictest5870-21apt20:50:133
2221399110,10cyclictest8826-21taskset20:55:213
2221399110,10cyclictest32226-21ssh00:01:203
2221399110,10cyclictest25012-21grep19:15:163
2221399110,0cyclictest0-21swapper/323:20:133
2220999119,1cyclictest28440-21tr19:20:272
2220999119,1cyclictest16093-21missed_timers00:15:252
2220999118,2cyclictest8902-21seq21:26:502
2220999118,2cyclictest7295-21ssh23:44:382
2220999118,2cyclictest6723-21cut23:20:182
2220999118,2cyclictest5810-21sed22:55:332
2220999118,2cyclictest32678-21grep19:30:162
2220999118,2cyclictest31425-21ssh23:15:082
2220999118,2cyclictest30860-21sh00:00:262
2220999118,2cyclictest27887-21ssh21:40:182
2220999118,2cyclictest25386-21grep20:20:342
2220999118,2cyclictest21409-21tr20:15:132
2220999118,2cyclictest21142-21ssh23:54:202
2220999118,2cyclictest17631-21cat21:10:332
2220999118,2cyclictest16610-21sh21:32:082
2220999118,2cyclictest15341-21diskmemload00:24:032
2220999118,2cyclictest11003-21head00:10:322
2220999118,2cyclictest10377-21basename21:00:002
2220999115,2cyclictest224-21systemd-journal20:12:272
2220999114,3cyclictest224-21systemd-journal20:05:002
2220999112,7cyclictest23186-21grep23:55:202
2220999111,3cyclictest0-21swapper/222:50:312
2220999111,3cyclictest0-21swapper/222:50:302
2220599119,1cyclictest277402sleep121:17:501
2220599118,2cyclictest8951-21head23:45:211
2220599118,2cyclictest763-21ssh00:30:091
2220599118,2cyclictest7391-21head22:10:181
2220599118,2cyclictest6336-21sed21:25:201
2220599118,2cyclictest5857-21ssh21:46:301
2220599118,2cyclictest4803-21rt-features20:45:271
2220599118,2cyclictest3344-21ssh22:07:131
2220599118,2cyclictest28922-21sed19:20:321
2220599118,2cyclictest27639-21ssh21:40:101
2220599118,2cyclictest25553-21ssh21:38:291
2220599118,2cyclictest24915-21ssh22:00:071
2220599118,2cyclictest2327-21cut20:40:281
2220599118,2cyclictest1964-21grep22:30:141
2220599118,2cyclictest17697-21ssh23:28:461
2220599118,2cyclictest17047-21grep22:40:221
2220599118,2cyclictest16314-21head00:15:301
2220599117,2cyclictest8210-21/usr/sbin/munin20:55:171
2220599111,9cyclictest0-21swapper/121:02:171
2220399118,2cyclictest9840-21head19:50:200
2220399118,2cyclictest9178-21expr22:35:170
2220399118,2cyclictest5046-21head19:40:190
2220399118,2cyclictest31427-21tr20:35:150
2220399118,2cyclictest30854-21chrt00:00:250
2220399118,2cyclictest30379-21ssh22:26:360
2220399118,2cyclictest26856-21tune2fs20:25:170
2220399118,2cyclictest24173-21wc22:45:270
2220399118,2cyclictest2355-21ssh23:16:470
2220399118,2cyclictest21591-21sed00:20:190
2220399118,2cyclictest16346-21cat23:50:260
2220399118,2cyclictest10295-21ssh23:45:450
2220399118,2cyclictest10024-21chrt20:56:010
2220399117,2cyclictest3145-21grep19:35:260
2220399112,5cyclictest21202-21/usr/sbin/munin21:35:210
2221399108,1cyclictest670-21ntpq19:30:233
2221399108,1cyclictest4623-21ls19:40:163
2221399108,1cyclictest3036-21ntpq19:35:243
2221399108,1cyclictest19874-21expr23:30:153
2221399108,1cyclictest16980-21grep20:05:253
2221399108,1cyclictest13417-21cpu21:05:173
2221399107,2cyclictest15454-21cat20:00:323
2221399107,2cyclictest13809-21wc20:00:003
2221399100,9cyclictest20428-21vmstat20:10:363
2220999108,1cyclictest0-21swapper/220:45:172
2220599108,1cyclictest9009-21taskset19:50:031
2220599108,1cyclictest30790-21taskset20:33:571
2220599108,1cyclictest25017-21cat19:15:161
2220599108,1cyclictest23548-21cut19:10:241
2220599108,1cyclictest23169-21taskset20:16:101
2220599108,1cyclictest19417-21cut23:30:121
2220599107,2cyclictest26257-21grep20:25:121
2220599107,2cyclictest17680-21ssh23:05:121
2220599102,5cyclictest1051-21/usr/sbin/munin23:40:341
2220399108,1cyclictest540-21load19:30:210
2220399108,1cyclictest22803-21cat23:55:150
2220399107,2cyclictest7015-21cat19:45:160
2220399107,2cyclictest6588-21ssh22:56:190
2220399107,2cyclictest24314-21df_abs20:20:180
2220399107,2cyclictest1168-21aten2_r7power_e20:40:150
2220399107,2cyclictest1007-21ssh23:40:080
222059997,1cyclictest1976-21cut19:35:121
222039997,1cyclictest4399-21cat20:45:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional