You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 02:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 13, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9134210390,9sleep30-21swapper/307:08:183
8962210190,6sleep10-21swapper/107:06:031
926128268,9sleep00-21swapper/007:09:560
897828164,13sleep20-21swapper/207:06:162
93362740,0sleep00-21swapper/009:42:010
5902600,0sleep10-21swapper/112:35:171
136652500,0sleep20-21swapper/209:19:072
935499280,24cyclictest0-21swapper/212:34:432
935499280,24cyclictest0-21swapper/212:34:432
935499250,25cyclictest0-21swapper/212:07:082
935499250,25cyclictest0-21swapper/210:37:252
9358992321,2cyclictest39-21ksoftirqd/309:25:013
9354992320,2cyclictest516-21gdbus10:25:242
935499220,21cyclictest0-21swapper/209:23:152
935499220,19cyclictest0-21swapper/210:59:152
934599220,21cyclictest0-21swapper/011:45:150
935899210,2cyclictest0-21swapper/307:35:143
935499210,20cyclictest0-21swapper/209:31:272
935899201,1cyclictest0-21swapper/312:20:493
935499200,20cyclictest0-21swapper/210:51:342
9358991916,2cyclictest516-21gdbus11:00:153
935899191,4cyclictest1279-21grep09:35:233
934599195,7cyclictest27482-21munin-run08:55:000
934599190,18cyclictest0-21swapper/009:36:570
934599190,0cyclictest0-21swapper/007:30:470
935899184,3cyclictest141rcu_preempt10:40:523
935899180,0cyclictest141rcu_preempt11:57:143
9354991817,1cyclictest17643-21kworker/2:212:10:092
935499180,9cyclictest0-21swapper/207:20:142
934599182,8cyclictest15775-21apache_accesses08:30:130
9354991711,5cyclictest10491-21kworker/u8:111:30:272
9352991714,2cyclictest516-21gdbus10:50:241
9345991712,4cyclictest29342-21runrttasks09:31:250
9358991615,1cyclictest39-21ksoftirqd/309:56:103
9358991612,3cyclictest1986-21ssh12:35:263
935499166,1cyclictest141rcu_preempt09:40:592
935499164,12cyclictest0-21swapper/209:10:142
935499164,12cyclictest0-21swapper/209:10:132
935499160,1cyclictest0-21swapper/212:04:222
935499160,1cyclictest0-21swapper/210:46:502
935499160,15cyclictest0-21swapper/210:01:062
9345991612,3cyclictest13-21ksoftirqd/010:15:210
935899158,2cyclictest1155-21apt-get11:20:143
935899156,9cyclictest39-21ksoftirqd/307:40:153
935899153,10cyclictest381rcuc/310:40:003
9358991511,3cyclictest14369-21sh09:20:003
935899150,8cyclictest28733-21runrttasks11:40:503
9354991512,2cyclictest0-21swapper/208:45:262
9354991510,3cyclictest1044-21wc09:05:212
935499150,15cyclictest13435-21kworker/u8:210:16:172
935499150,15cyclictest0-21swapper/212:39:162
9352991511,3cyclictest17946-21sh08:34:591
9345991515,0cyclictest0-21swapper/012:13:010
9345991512,2cyclictest13-21ksoftirqd/007:20:170
934599150,1cyclictest0-21swapper/010:11:090
935899145,1cyclictest2209-21diskmemload11:34:503
935899141,6cyclictest0-21swapper/307:25:123
9358991412,2cyclictest39-21ksoftirqd/311:35:133
935499149,2cyclictest27982-21tr08:55:142
9354991411,2cyclictest28663-21taskset11:40:452
9354991411,2cyclictest18643-21wc12:25:002
9354991410,3cyclictest14276-21sed11:55:252
935499140,13cyclictest0-21swapper/210:10:142
935499140,13cyclictest0-21swapper/210:06:002
935499140,12cyclictest0-21swapper/209:00:152
9352991411,2cyclictest5762-21apt08:10:001
935299140,14cyclictest0-21swapper/111:26:011
9345991414,0cyclictest0-21swapper/011:39:310
9345991411,2cyclictest6279-21ntp_kernel_pll_12:15:220
934599140,14cyclictest0-21swapper/009:54:270
935899139,3cyclictest27468-21cron08:55:003
935899139,3cyclictest22879-21head10:45:243
935899137,2cyclictest39-21ksoftirqd/312:17:193
935899133,5cyclictest224-21systemd-journal08:10:003
9358991311,1cyclictest496-21dbus-daemon10:30:253
9358991311,1cyclictest39-21ksoftirqd/311:49:143
9358991310,2cyclictest30380-21tr09:00:163
9358991310,2cyclictest29940-21cat09:00:003
9358991310,2cyclictest20066-21sed12:00:183
9358991310,2cyclictest19377-21grep09:50:163
9358991310,2cyclictest11199-21grep07:10:283
9358991310,2cyclictest10517-21basename10:10:153
935899130,13cyclictest0-21swapper/310:25:253
935499139,3cyclictest8438-21wc11:50:362
935499133,5cyclictest224-21systemd-journal08:34:592
9354991311,1cyclictest21797-21wc11:35:282
9354991310,2cyclictest6575-21wc11:50:002
9354991310,2cyclictest6474-21tr08:10:162
9354991310,2cyclictest5105-21tr12:15:142
9354991310,2cyclictest29438-21grep07:50:182
9354991310,2cyclictest28438-21cron07:50:012
935499130,2cyclictest0-21swapper/209:37:512
935499130,11cyclictest0-21swapper/211:13:562
935299139,3cyclictest27371-21wc12:30:271
935299139,3cyclictest27371-21wc12:30:261
935299139,3cyclictest26669-21tr12:05:241
935299139,3cyclictest18488-21head07:25:251
935299138,3cyclictest9984-21wc11:00:291
935299138,2cyclictest7788-21grep11:00:001
935299135,7cyclictest13912-21sh12:20:261
9352991312,1cyclictest7744-21snmpd07:47:501
9352991310,2cyclictest22990-21grep10:20:161
9352991310,2cyclictest22168-21wc11:10:201
9352991310,2cyclictest20066-2110-uname11:09:211
9352991310,2cyclictest19852-21tail08:35:281
9352991310,2cyclictest0-21swapper/109:05:151
935299130,11cyclictest0-21swapper/110:18:011
934599134,8cyclictest14089-21munin-run07:20:010
934599134,8cyclictest14089-21munin-run07:20:000
9345991310,2cyclictest8101-21tail11:25:200
9345991310,2cyclictest27846-21cat07:45:260
9345991310,2cyclictest21855-21cut10:45:160
9345991310,2cyclictest11262-21tr08:20:130
9345991310,2cyclictest10386-21grep10:35:240
935899128,3cyclictest29426-21grep07:50:183
935899127,4cyclictest29624-21kill11:15:513
935899127,4cyclictest2209-21diskmemload12:08:033
935899126,5cyclictest39-21ksoftirqd/309:40:183
935899126,1cyclictest141rcu_preempt12:33:413
935899126,1cyclictest141rcu_preempt12:33:403
935899120,2cyclictest0-21swapper/309:46:113
935899120,1cyclictest0-21swapper/310:54:053
935499129,2cyclictest2512-21head11:20:242
935499129,2cyclictest24947-21grep09:55:152
935499129,2cyclictest21823-21grep07:35:152
935499129,2cyclictest21315-21basename07:35:012
935499129,2cyclictest16872-21cut07:25:122
935499129,2cyclictest10327-21sh11:26:432
935499122,2cyclictest493-21systemd-logind11:00:152
9354991211,1cyclictest0-21swapper/210:40:422
9354991210,1cyclictest20696-21cron11:10:012
935299129,2cyclictest972-21sh08:00:001
935299129,2cyclictest8603-21ls10:34:591
935299129,2cyclictest7172-21grep11:50:151
935299129,2cyclictest703-21basename11:20:011
935299129,2cyclictest32363-21cut12:10:181
935299129,2cyclictest32055-21awk09:35:011
935299129,2cyclictest30563-21awk10:00:001
935299129,2cyclictest28946-21grep07:50:151
935299129,2cyclictest2869-21grep09:10:151
935299129,2cyclictest2869-21grep09:10:141
935299129,2cyclictest2305-21wc08:00:201
935299129,2cyclictest21817-21grep07:35:151
935299129,2cyclictest19827-21df12:00:171
935299129,2cyclictest11896-21awk10:10:231
935299129,2cyclictest11273-21systemctl12:20:001
935299129,2cyclictest10731-21cron08:19:591
935299128,3cyclictest9426-21grep09:15:171
935299128,3cyclictest31196-21wc09:00:221
935299124,5cyclictest16439-21ntpq10:40:291
935299121,6cyclictest0-21swapper/110:25:141
934599129,2cyclictest698-21munin-plugin-st11:20:010
934599129,2cyclictest4418-21grep09:10:260
934599129,2cyclictest4418-21grep09:10:250
934599129,2cyclictest32165-21cron12:35:010
934599129,2cyclictest32165-21cron12:35:000
934599129,2cyclictest30473-21tr10:25:230
934599129,2cyclictest25189-21cat12:30:010
934599129,2cyclictest22325-21grep07:35:180
934599129,2cyclictest21156-21ssh11:10:140
934599129,2cyclictest2024-21head08:00:180
934599129,2cyclictest18882-21cat11:59:590
934599129,2cyclictest16793-21grep09:20:270
934599129,2cyclictest13662-21tr08:25:170
934599128,3cyclictest31124-21cat09:00:190
934599120,11cyclictest18555-21ssh12:24:590
935899119,1cyclictest21499-21head09:25:193
935899119,1cyclictest16531-21cron07:25:003
935899118,3cyclictest39-21ksoftirqd/310:16:333
935899118,2cyclictest9489-21ssh11:25:503
935899118,2cyclictest6071-21ssh09:12:293
935899118,2cyclictest6071-21ssh09:12:283
935899118,2cyclictest4289-21ssh10:05:043
935899118,2cyclictest39-21ksoftirqd/307:20:013
935899118,2cyclictest39-21ksoftirqd/307:20:013
935899118,2cyclictest3374-21cron08:04:593
935899118,2cyclictest32588-21tr09:05:163
935899118,2cyclictest31442-21expr10:00:153
935899118,2cyclictest28077-21grep09:30:233
935899118,2cyclictest24087-21sh11:11:453
935899118,2cyclictest21305-21ssh12:25:553
935899118,2cyclictest18121-21cat08:35:003
935899118,2cyclictest15463-21head11:05:203
935899118,1cyclictest39-21ksoftirqd/308:40:183
935899116,4cyclictest39-21ksoftirqd/308:20:223
935899110,10cyclictest809-21wc12:10:273
935499119,1cyclictest25570-21sed07:40:272
935499118,2cyclictest8906-21tr08:15:142
935499118,2cyclictest6927-21ssh10:32:412
935499118,2cyclictest37850irq/131-snd_hda08:35:162
935499118,2cyclictest25584-21tr08:50:152
935499118,2cyclictest25098-21ssh09:28:282
935499118,2cyclictest23770-21sh10:20:192
935499118,2cyclictest23272-21ssh12:27:542
935499118,2cyclictest21600-21wc08:40:202
935499118,2cyclictest20628-21ssh09:50:282
935499118,2cyclictest1471-21tr08:00:142
935499118,2cyclictest14086-21munin-plugin-st07:20:012
935499118,2cyclictest14086-21munin-plugin-st07:20:012
935499118,2cyclictest12089-21ssh09:45:012
935499118,2cyclictest11099-21cat07:10:272
935499116,3cyclictest27388-21apache_accesses11:15:122
9354991111,0cyclictest0-21swapper/208:20:132
935299118,2cyclictest6417-21cat09:40:161
935299118,2cyclictest4944-21ssh10:05:181
935299118,2cyclictest3590-21taskset11:21:111
935299118,2cyclictest2897-21ssh10:03:331
935299118,2cyclictest28912-21ssh11:41:011
935299118,2cyclictest26104-21chrt09:29:361
935299118,2cyclictest23639-21ssh10:45:411
935299118,2cyclictest21603-21rm12:26:131
935299118,2cyclictest20698-21df11:35:171
935299118,2cyclictest20626-21basename08:40:161
935299118,2cyclictest15964-21ssh11:30:561
935299118,2cyclictest15178-21cat09:20:151
935299118,2cyclictest1475-21wc09:35:241
935299118,2cyclictest12931-21ssh09:45:181
935299118,2cyclictest12771-21ssh10:37:371
935299118,2cyclictest12610-21sed08:20:261
935299118,2cyclictest10694-21strings07:10:221
935299117,3cyclictest13612-21sed07:15:271
935299117,3cyclictest13612-21sed07:15:261
935299117,2cyclictest25613-21grep08:50:151
935299113,6cyclictest19184-21apt-get07:30:121
934599119,1cyclictest32392-21tr07:55:200
934599118,2cyclictest9064-21cat08:15:160
934599118,2cyclictest7584-21grep11:50:180
934599118,2cyclictest6570-21ssh11:24:420
934599118,2cyclictest3465-21ssh10:04:190
934599118,2cyclictest30794-21taskset07:54:480
934599118,2cyclictest27772-21ssh12:06:060
934599118,2cyclictest26857-21expr11:40:150
934599118,2cyclictest24673-21ssh09:55:140
934599118,2cyclictest21108-21grep12:00:280
934599118,2cyclictest20593-21ssh09:25:080
934599118,2cyclictest18718-21sh11:33:560
934599118,2cyclictest1783-21taskset10:55:160
934599118,2cyclictest16136-21ssh09:47:380
934599118,2cyclictest15981-21sed10:40:260
934599118,2cyclictest12234-21ssh09:17:170
934599117,3cyclictest2325-21grep12:35:290
934599111,2cyclictest224-21systemd-journal08:09:590
935899108,1cyclictest8615-21chrt08:15:123
935899108,1cyclictest22766-21sh10:20:093
935899108,1cyclictest2015-21expr10:55:163
935899108,1cyclictest19258-21cat08:35:223
935899107,2cyclictest7363-21ntpq08:10:213
935899107,2cyclictest7209-21tr11:50:153
935899107,2cyclictest657-21taskset07:56:023
935899107,2cyclictest26603-21tr07:45:143
935899107,2cyclictest19942-21head07:30:163
935899107,2cyclictest15418-21taskset08:28:553
935899100,9cyclictest22988-21apache_processe08:45:153
935499107,2cyclictest15084-21grep08:25:282
935499106,4cyclictest16096-21apache207:58:072
935299108,1cyclictest18622-21ssh09:50:071
935299108,1cyclictest15745-21chrt11:56:371
935299108,1cyclictest15367-21chrt08:28:171
935299107,2cyclictest6331-21basename08:10:151
935299107,2cyclictest545-21ssh11:45:111
935299107,2cyclictest24935-21chrt08:48:041
935299107,2cyclictest24224-21grep07:40:161
935299107,2cyclictest14476-21cut07:20:141
935299100,9cyclictest29003-21processes08:55:211
934599108,2cyclictest29182-21ntpq10:50:220
934599108,1cyclictest7744-21snmpd07:12:420
934599108,1cyclictest6189-21appstreamcli08:10:010
934599108,1cyclictest4270-21sh10:05:020
934599108,1cyclictest295212sleep008:55:530
934599108,1cyclictest24285-21cat07:40:160
934599108,1cyclictest23471-21df08:45:170
934599108,1cyclictest14477-21ssh11:05:040
934599108,1cyclictest10753-21ssh11:01:060
934599107,2cyclictest23167-21tr10:20:160
934599107,2cyclictest21306-21tune2fs08:40:190
934599107,2cyclictest18312-21cat07:25:230
934599100,9cyclictest3090-21cut10:30:150
934599100,9cyclictest181692sleep008:35:010
93549992,6cyclictest0-21swapper/208:05:162
93459996,2cyclictest32745-21grep09:05:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional