You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-22 - 12:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jan 22, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6175210188,9sleep10-21swapper/119:09:521
5965210082,10sleep30-21swapper/319:07:103
600628363,16sleep20-21swapper/219:07:402
600428166,9sleep00-21swapper/019:07:380
229732340,1chrt224-21systemd-journal19:45:010
628199300,29cyclictest0-21swapper/300:32:343
628199263,2cyclictest21279-21grep20:50:133
628199250,1cyclictest0-21swapper/323:28:083
628199250,1cyclictest0-21swapper/300:01:043
627399250,23cyclictest0-21swapper/021:33:080
6281992423,1cyclictest0-21swapper/321:18:593
628199240,1cyclictest141rcu_preempt22:15:123
628199234,10cyclictest0-21swapper/323:30:093
628199231,4cyclictest224-21systemd-journal23:15:003
628199212,1cyclictest0-21swapper/300:06:103
628199210,7cyclictest21400-21grep22:20:203
628199210,21cyclictest0-21swapper/322:02:523
627899210,20cyclictest0-21swapper/221:30:222
628199200,3cyclictest0-21swapper/320:30:183
628199200,0cyclictest0-21swapper/321:39:273
628199192,7cyclictest29782-21diskmemload21:30:103
6278991916,1cyclictest0-21swapper/200:05:002
627899190,4cyclictest0-21swapper/221:26:422
627899190,16cyclictest0-21swapper/223:34:452
627599199,9cyclictest9271-21apache222:02:411
628199184,2cyclictest21106-21latency_hist00:25:003
628199182,15cyclictest12673-21sh21:46:573
627899180,16cyclictest0-21swapper/223:15:562
627899180,14cyclictest0-21swapper/223:10:202
627399184,8cyclictest31190-21awk22:55:000
6273991813,4cyclictest14312-21ssh00:19:450
628199175,3cyclictest224-21systemd-journal20:00:203
628199172,7cyclictest13981-21chrt00:19:243
628199171,2cyclictest0-21swapper/323:04:513
6281991710,6cyclictest20628-21taskset23:59:553
627899179,6cyclictest6441-21/usr/sbin/munin21:40:272
6278991716,1cyclictest0-21swapper/221:35:402
627899170,17cyclictest0-21swapper/222:59:512
628199169,7cyclictest0-21swapper/323:09:513
628199169,6cyclictest17922-21turbostat21:25:013
628199169,6cyclictest15883-21head20:35:183
628199169,2cyclictest39-21ksoftirqd/323:18:503
628199162,12cyclictest16105-21apt-get19:30:163
6281991610,5cyclictest1017-21sed22:55:253
6278991610,5cyclictest25368-21basename19:50:012
6275991615,1cyclictest0-21swapper/123:03:041
6273991616,0cyclictest0-21swapper/019:35:140
628199159,5cyclictest23573-21grep20:55:143
628199159,5cyclictest11142-21cut20:25:173
628199158,2cyclictest5392-21grep21:40:183
628199158,2cyclictest32020-21awk00:35:203
628199158,1cyclictest31383-21cat21:10:263
628199158,1cyclictest25226-21chrt19:48:323
628199157,2cyclictest27806-21ls19:55:003
628199157,2cyclictest27806-21ls19:55:003
628199151,4cyclictest21092-21grep19:40:153
6281991514,1cyclictest0-21swapper/322:06:193
628199150,8cyclictest4479-21sh22:33:433
628199150,1cyclictest0-21swapper/319:25:143
6278991513,2cyclictest24758-21kworker/2:221:20:112
6278991513,1cyclictest61950irq/127-enp2s0-23:46:212
627899150,15cyclictest0-21swapper/223:56:012
627599151,5cyclictest224-21systemd-journal20:20:001
6275991511,2cyclictest18278-21wc19:35:001
627599150,14cyclictest0-21swapper/123:38:321
627599150,13cyclictest29785-21awk21:10:011
6273991512,2cyclictest2671-21cron20:10:000
628199148,2cyclictest6479-21cat22:35:143
628199148,1cyclictest30546-21sh23:41:583
628199148,1cyclictest21279-21ssh21:28:053
628199147,2cyclictest6952-21expr19:10:163
628199147,2cyclictest5469-21sh23:23:553
628199147,2cyclictest3104-21chrt23:46:413
628199147,2cyclictest23309-21cat21:55:183
628199147,2cyclictest21748-21ssh00:25:433
628199147,2cyclictest21312-21ssh23:35:233
628199146,2cyclictest26918-21tail21:00:253
628199145,2cyclictest39-21ksoftirqd/322:50:493
628199141,1cyclictest0-21swapper/323:52:363
627899144,9cyclictest14967-21apache_accesses00:20:122
6278991411,2cyclictest6181-21grep22:35:132
6278991410,3cyclictest25233-21ssh22:23:212
627599149,3cyclictest22072-21cut00:00:161
6275991410,2cyclictest26865-21wc23:15:261
627599140,4cyclictest11096-21basename19:20:001
627399146,4cyclictest25757-21ssh21:56:580
627399143,7cyclictest30824-31find00:07:530
6273991410,2cyclictest15264-21grep00:20:150
628199138,1cyclictest21486-21ssh22:46:373
628199138,1cyclictest20537-21taskset19:39:353
628199138,1cyclictest20093-21chrt20:48:153
628199138,1cyclictest16095-21cat21:50:163
628199138,1cyclictest109372sleep319:18:353
628199138,1cyclictest10061-21ssh22:10:443
628199137,1cyclictest5356-21taskset00:12:033
628199137,1cyclictest13746-21cat22:40:143
627899139,3cyclictest9636-21grep22:10:212
627899139,3cyclictest4130-21ls21:15:002
6278991311,2cyclictest29782-21diskmemload23:38:042
6278991310,2cyclictest29165-21cat19:55:192
6278991310,2cyclictest26332-21grep19:50:202
6278991310,2cyclictest26332-21grep19:50:202
6278991310,2cyclictest20090-21grep19:35:272
6278991310,2cyclictest17247-21munin-run20:40:002
6278991310,2cyclictest12094-21cat23:30:012
627899130,11cyclictest0-21swapper/200:18:422
627599139,3cyclictest15230-21grep22:40:261
627599139,3cyclictest13236-21grep23:30:131
627599138,3cyclictest5259-21sh20:15:001
627599138,3cyclictest0-21swapper/122:33:471
627599137,6cyclictest31862-21apache200:09:531
627599137,3cyclictest0-21swapper/123:20:111
627599135,7cyclictest9279-21apache223:11:071
6275991310,2cyclictest8632-21grep00:15:141
6275991310,2cyclictest4873-21tr21:40:151
6275991310,2cyclictest25983-21grep21:00:151
6275991310,2cyclictest2552-21cut00:10:121
6275991310,2cyclictest23805-21uname20:55:181
6275991310,2cyclictest1753-21sed20:05:211
6275991310,2cyclictest16667-21cut21:50:191
6275991310,2cyclictest15097-21tail22:15:181
627599130,12cyclictest16180-21head00:20:201
627399133,3cyclictest31959-21switchtime21:10:320
627399132,5cyclictest12262-21dump-pmu-power23:05:000
627399131,4cyclictest0-21swapper/019:10:150
6273991310,2cyclictest8676-21grep23:50:210
6273991310,2cyclictest4872-21ssh23:23:190
6273991310,2cyclictest28044-21dump-pmu-power00:05:000
6273991310,2cyclictest19049-21sort23:10:160
627399130,12cyclictest31200-21grep21:35:160
627399130,12cyclictest0-21swapper/022:10:420
628199129,2cyclictest3900-21grep20:10:203
628199129,2cyclictest19134-21grep20:40:273
628199129,2cyclictest12097-21cut19:20:163
628199128,1cyclictest27164-21sh22:25:103
628199127,1cyclictest7286-21taskset20:15:373
627899129,2cyclictest6783-21tr19:10:152
627899129,2cyclictest22971-21cron19:45:002
627899129,2cyclictest22409-21ssh21:55:132
627899129,2cyclictest16527-21grep21:50:182
627899129,2cyclictest1641-21sed23:20:232
627899129,2cyclictest13963-21tr19:25:142
627899129,2cyclictest12490-21sleep20:30:002
627899128,3cyclictest30496-21sh22:54:192
627899128,3cyclictest27330-21tr23:40:132
627899128,3cyclictest22711-21cat20:55:002
627899122,1cyclictest25542-21cpu00:30:102
6278991210,1cyclictest30815-21systemd-cgroups00:07:232
627899120,11cyclictest0-21swapper/222:32:002
627599129,2cyclictest31064-21tr00:35:141
627599129,2cyclictest29434-21grep19:55:221
627599129,2cyclictest25612-21cut00:30:101
627599129,2cyclictest23168-21tail21:55:161
627599129,2cyclictest18737-21head20:40:231
627599129,2cyclictest17394-21wc20:40:001
627599129,2cyclictest12790-21grep21:20:181
627599129,2cyclictest12008-21chrt22:12:591
627599129,2cyclictest10979-21cron21:20:011
627599129,2cyclictest1006-21tail22:55:251
627599128,3cyclictest13975-21tr19:25:141
627599128,2cyclictest6936-21cut22:35:161
627599128,2cyclictest22109-21grep20:50:221
627599121,9cyclictest0-21swapper/122:20:211
6275991211,1cyclictest0-21swapper/119:20:131
627599121,10cyclictest9473-21munin-node23:43:341
627399129,2cyclictest6274-21sed20:15:170
627399129,2cyclictest3839-21grep00:40:000
627399129,2cyclictest31152-21grep20:00:170
627399129,2cyclictest29285-21grep22:00:150
627399129,2cyclictest29266-21sed21:05:280
627399129,2cyclictest25828-21grep21:00:140
627399129,2cyclictest19181-21wc23:35:000
627399129,2cyclictest16380-21wc20:35:220
627399129,2cyclictest16127-21grep21:50:170
627399129,2cyclictest13-21ksoftirqd/022:15:200
627399129,2cyclictest12255-21grep21:20:140
627399128,3cyclictest31583-21turbostat.cron22:30:010
627399128,3cyclictest20753-21basename00:00:000
627399127,3cyclictest20236-21cron20:49:590
627399127,3cyclictest10562-21sed19:15:270
627399125,3cyclictest5864-21ntp_states21:40:230
627399123,5cyclictest224-21systemd-journal20:44:590
627399121,1cyclictest141rcu_preempt23:28:440
627399120,0cyclictest0-21swapper/023:47:190
628199118,2cyclictest8995-21sort20:20:213
628199118,2cyclictest2660-21sh20:10:013
627899118,2cyclictest8870-21basename19:15:132
627899118,2cyclictest8427-21ssh23:50:192
627899118,2cyclictest6292-21cut20:15:172
627899118,2cyclictest4764-21ssh22:07:062
627899118,2cyclictest4560-21ssh00:11:122
627899118,2cyclictest32316-21grep20:00:272
627899118,2cyclictest3135-21basename20:10:142
627899118,2cyclictest27919-21cut21:05:152
627899118,2cyclictest25994-21head21:00:152
627899118,2cyclictest23627-21grep20:55:152
627899118,2cyclictest23563-21ssh22:48:532
627899118,2cyclictest22910-21ssh00:27:012
627899118,2cyclictest17227-21ssh22:42:252
627899118,2cyclictest16511-21cat19:30:172
627899118,2cyclictest15526-21cut22:15:202
627899118,2cyclictest14301-21cat20:30:302
627899118,2cyclictest13962-21ssh23:06:292
627899118,2cyclictest10779-21ssh21:19:442
627899110,3cyclictest0-21swapper/222:29:102
627899110,11cyclictest0-21swapper/200:35:152
627899110,10cyclictest10935-21sh21:45:012
627599119,1cyclictest6470-21cron23:50:001
627599118,2cyclictest8726-21grep23:50:211
627599118,2cyclictest7494-21wc19:10:201
627599118,2cyclictest31574-21cron22:30:001
627599118,2cyclictest30623-21cat21:10:211
627599118,2cyclictest2932-21ssh22:05:011
627599118,2cyclictest25908-21tr19:50:191
627599118,2cyclictest25908-21tr19:50:181
627599118,2cyclictest19380-21ssh21:26:131
627599118,2cyclictest14882-21wc20:34:591
627599118,2cyclictest1310-21ssh21:36:511
627599118,2cyclictest12250-21sh21:46:271
627599117,2cyclictest21131-21tr19:40:151
627599113,6cyclictest29782-21diskmemload23:08:581
627599111,5cyclictest8290-21ssh23:26:251
627599110,7cyclictest0-21swapper/120:00:141
627599110,10cyclictest27453-21ssh21:32:161
627399118,2cyclictest8081-21egrep20:20:170
627399118,2cyclictest6554-21expr22:35:140
627399118,2cyclictest4300-21ntpq20:10:250
627399118,2cyclictest3906-21ssh22:33:050
627399118,2cyclictest31551-21ssh23:42:570
627399118,2cyclictest30225-21chrt19:58:580
627399118,2cyclictest27494-21ssh00:30:470
627399118,2cyclictest27179-21rm23:15:340
627399118,2cyclictest24397-21sed20:55:240
627399118,2cyclictest2431-21ssh22:56:480
627399118,2cyclictest22155-21ssh23:35:510
627399118,2cyclictest21251-21ssh21:28:030
627399118,2cyclictest20756-21tr20:50:110
627399118,2cyclictest20516-21ssh22:45:350
627399118,2cyclictest2026-21grep00:10:100
627399118,2cyclictest19027-21ssh22:44:170
627399118,2cyclictest14370-21cut19:25:160
627399118,2cyclictest12249-21cat19:20:190
627399118,2cyclictest11682-21ssh21:45:500
627399115,5cyclictest472-21avahi-daemon23:09:470
627399113,7cyclictest18226-21awk19:35:010
627399110,10cyclictest6987-21ssh21:15:530
627399110,10cyclictest5558-21ssh22:08:020
280192111,5sleep128029-21ssh22:51:391
628199108,1cyclictest28884-21grep21:05:223
628199107,2cyclictest28625-21expr19:55:163
627899108,1cyclictest29229-21cat22:00:152
627899108,1cyclictest12464-21ntp_kernel_err19:20:192
627899107,2cyclictest7909-21cat20:20:132
627899107,2cyclictest6337-21cat23:00:162
627899107,2cyclictest18522-21cat20:40:202
627899103,3cyclictest224-21systemd-journal20:50:002
627599108,1cyclictest204302sleep119:38:121
627599108,1cyclictest19969-21taskset20:46:371
627599107,2cyclictest21494-21ssh00:25:231
627599107,2cyclictest16598-21ssh23:55:581
627599107,2cyclictest11785-21grep20:25:241
627599102,5cyclictest0-21swapper/122:45:091
627399108,1cyclictest23600-21cat19:45:140
627399107,2cyclictest26582-21wc19:50:230
627399107,2cyclictest26582-21wc19:50:220
627399107,2cyclictest21289-21ssh00:25:100
627399107,2cyclictest21285-21grep22:20:180
627399107,2cyclictest13012-21grep20:30:200
627399107,2cyclictest11447-21taskset20:25:190
62789997,1cyclictest1581-21wc20:05:192
62759997,1cyclictest251622sleep119:47:411
62759996,2cyclictest8020-21cut20:20:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional