You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 12:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 08, 2025 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27090210075,9sleep10-21swapper/119:08:281
26993210075,9sleep30-21swapper/319:07:143
2692228564,6sleep20-21swapper/219:06:202
2706228169,8sleep00-21swapper/019:08:070
308382550,0sleep20-21swapper/221:17:032
2730499300,29cyclictest0-21swapper/223:19:372
2730499270,27cyclictest0-21swapper/223:41:542
2730499260,25cyclictest0-21swapper/223:23:232
2730499210,21cyclictest0-21swapper/200:18:012
2730499210,21cyclictest0-21swapper/200:18:002
2729799212,1cyclictest0-21swapper/022:01:390
2729799210,1cyclictest0-21swapper/023:45:200
2730499204,16cyclictest19936-21diskmemload22:24:472
27304992019,1cyclictest27277-21apt-get22:00:142
2729799200,2cyclictest0-21swapper/021:49:110
27307991917,2cyclictest39-21ksoftirqd/323:25:173
2730799190,2cyclictest0-21swapper/321:26:123
2730499191,17cyclictest9365-21cpuspeed_turbos22:10:152
2730099196,12cyclictest0-21swapper/120:30:141
2730099190,15cyclictest0-21swapper/122:57:191
2729799190,19cyclictest0-21swapper/023:40:140
2730499180,18cyclictest0-21swapper/200:31:332
2730499180,14cyclictest0-21swapper/221:21:102
2730499180,12cyclictest0-21swapper/223:36:262
2730099186,9cyclictest28617-21runrttasks21:38:321
27300991817,1cyclictest0-21swapper/123:20:221
27300991814,1cyclictest0-21swapper/121:56:541
2729799183,4cyclictest0-21swapper/022:10:140
27297991817,1cyclictest0-21swapper/020:40:170
2730499173,13cyclictest19936-21diskmemload22:28:242
27300991714,2cyclictest25-21ksoftirqd/122:05:211
82682160,1sleep30-21swapper/321:24:373
27304991613,1cyclictest0-21swapper/200:00:222
2730499161,13cyclictest0-21swapper/223:31:452
27300991616,0cyclictest0-21swapper/123:03:011
2729799168,7cyclictest4372-21munin-run19:30:000
27297991611,4cyclictest30241-21sleep21:40:000
27304991513,1cyclictest0-21swapper/223:00:502
2730499150,2cyclictest0-21swapper/221:45:022
2730499150,15cyclictest0-21swapper/221:40:292
2730499150,14cyclictest0-21swapper/220:55:152
2730499150,10cyclictest0-21swapper/221:50:442
27300991511,2cyclictest15171-21grep20:59:591
2730099150,13cyclictest0-21swapper/121:51:201
2729799152,7cyclictest0-21swapper/020:30:140
2729799150,1cyclictest0-21swapper/020:20:170
2729799150,1cyclictest0-21swapper/020:20:160
2730799149,3cyclictest2422-21ssh22:50:173
27307991410,3cyclictest28085-21grep22:00:183
27307991410,3cyclictest26695-21sh22:45:013
27307991410,2cyclictest18877-21wc20:00:003
27307991410,2cyclictest17779-21grep23:00:283
2730799140,1cyclictest0-21swapper/323:09:333
2730499145,6cyclictest37850irq/131-snd_hda20:00:142
2730499143,4cyclictest516-21gdbus22:40:012
2730499141,2cyclictest28248-21cron20:20:002
27304991412,2cyclictest32-21ksoftirqd/223:26:082
2730499141,13cyclictest0-21swapper/221:10:152
2730499140,14cyclictest0-21swapper/222:45:502
27300991413,1cyclictest0-21swapper/100:01:561
27300991411,2cyclictest25-21ksoftirqd/123:38:161
2730099140,11cyclictest0-21swapper/122:43:231
2729799144,6cyclictest224-21systemd-journal20:25:010
2729799143,6cyclictest6463-21hddtemp_smartct22:30:190
27297991412,1cyclictest27292-21cyclictest23:34:110
2729799140,14cyclictest0-21swapper/022:48:400
2730799139,2cyclictest13000-21tr22:35:153
2730799133,5cyclictest224-21systemd-journal00:09:243
27307991310,2cyclictest9615-21wc00:30:013
27307991310,2cyclictest4527-21latency_hist00:00:003
27307991310,2cyclictest4527-21latency_hist00:00:003
27307991310,2cyclictest28274-21grep23:30:283
27307991310,2cyclictest19064-21cut20:00:143
27307991310,2cyclictest16676-21grep00:35:203
2730499133,8cyclictest19936-21diskmemload00:23:222
2730499133,10cyclictest0-21swapper/222:41:382
27304991313,0cyclictest0-21swapper/200:36:402
2730499131,11cyclictest19936-21diskmemload22:56:082
27304991310,2cyclictest22797-21apt23:50:002
2730499130,12cyclictest24078-21tail23:50:192
2730499130,12cyclictest1291-21runrttasks00:05:482
27300991310,2cyclictest9606-21grep23:40:181
27300991310,2cyclictest7335-21grep21:45:281
27300991310,2cyclictest32221-21sed23:55:281
27300991310,2cyclictest32221-21sed23:55:281
27300991310,2cyclictest32122-21ls22:25:231
27300991310,2cyclictest30075-21taskset22:46:241
27300991310,2cyclictest24039-21grep20:10:181
27300991310,2cyclictest13538-21grep19:45:261
2729799139,2cyclictest24079-21rm22:20:170
2729799136,4cyclictest23060-21dump-pmu-power21:35:000
2729799133,4cyclictest4530-21apt-get00:00:030
2729799132,1cyclictest0-21swapper/022:36:020
27297991310,1cyclictest3659-21expr20:35:170
2729799130,12cyclictest1688-21tr23:35:160
2730799129,3cyclictest39-21ksoftirqd/321:50:053
2730799129,2cyclictest7071-21grep00:00:293
2730799129,2cyclictest32150-21wc19:20:003
2730799129,2cyclictest27571-21tr21:15:163
2730799129,2cyclictest24843-21sed21:35:213
2730799129,2cyclictest23991-21grep20:10:183
2730799129,2cyclictest2283-21grep22:05:163
2730799129,2cyclictest22269-21tail21:10:283
2730799129,2cyclictest17657-21head21:30:223
2730799129,2cyclictest16226-21head21:00:183
2730799129,2cyclictest13927-21sed23:20:273
2730799129,2cyclictest12554-21cron20:55:013
2730799128,3cyclictest27791-21grep20:15:293
2730799123,1cyclictest224-21systemd-journal19:50:003
2730799122,9cyclictest19936-21diskmemload23:38:363
2730499129,2cyclictest20372-21sh00:10:322
2730499129,2cyclictest17321-21grep19:55:192
2730499129,2cyclictest15727-21grep21:00:152
2730499129,1cyclictest2727-21grep22:05:192
2730499128,3cyclictest8303-21sh00:28:412
2730499128,3cyclictest6852-21tr22:30:212
2730499124,4cyclictest224-21systemd-journal22:16:472
27304991211,1cyclictest37850irq/131-snd_hda21:27:232
27304991210,1cyclictest1535-21grep19:20:302
2730499120,10cyclictest18302-21tail21:30:292
2730099129,2cyclictest6688-21tail23:15:271
2730099129,2cyclictest5038-21cat21:45:001
2730099129,2cyclictest4476-21wc23:15:001
2730099129,2cyclictest3235-21wc22:50:231
2730099129,2cyclictest25425-21sed00:15:211
2730099129,2cyclictest25425-21sed00:15:201
2730099129,2cyclictest24192-21cron00:15:001
2730099129,2cyclictest24122-21sed23:50:191
2730099129,2cyclictest23243-21grep21:35:001
2730099129,2cyclictest19768-21head20:00:171
2730099129,2cyclictest15177-21cat00:35:011
2730099129,2cyclictest11685-21cat19:45:001
2730099128,3cyclictest9907-21grep22:10:181
2730099128,3cyclictest22348-21grep21:10:271
2730099128,3cyclictest17580-21cat21:05:001
2730099128,2cyclictest14239-21basename19:50:151
2730099123,5cyclictest224-21systemd-journal19:40:001
2730099122,8cyclictest11561-21apt-config00:05:011
27300991211,1cyclictest0-21swapper/123:27:271
2730099120,12cyclictest0-21swapper/100:27:561
2729799129,2cyclictest32619-21sed21:40:270
2729799129,2cyclictest32053-21grep23:10:260
2729799129,2cyclictest23456-21grep23:05:210
2729799129,2cyclictest20942-21sort23:25:290
2729799129,2cyclictest19874-21ssh23:02:250
2729799129,2cyclictest18969-21ssh00:37:310
2729799129,2cyclictest18107-21grep22:15:280
2729799129,2cyclictest17595-21ssh00:09:440
2729799129,2cyclictest1537-21sed19:20:300
2729799129,2cyclictest13550-21sed19:45:260
2729799129,2cyclictest12359-21grep23:20:150
2729799129,2cyclictest12119-21sed20:50:280
2729799128,2cyclictest0-21swapper/022:55:150
2729799124,6cyclictest0-21swapper/021:10:140
2729799121,4cyclictest224-21systemd-journal00:29:590
27297991211,1cyclictest0-21swapper/000:23:030
27297991210,1cyclictest10247-21tail19:40:180
2729799120,1cyclictest0-21swapper/021:21:100
2729799120,1cyclictest0-21swapper/020:55:150
2729799120,11cyclictest15099-21cron00:35:010
16702120,0sleep30-21swapper/321:41:493
2730799119,1cyclictest25445-21grep22:20:283
2730799119,1cyclictest116642sleep322:11:013
2730799118,2cyclictest9917-21ssh00:30:123
2730799118,2cyclictest9759-21expr19:40:153
2730799118,2cyclictest970-21ssh22:26:053
2730799118,2cyclictest9217-21tr22:55:163
2730799118,2cyclictest8709-21ssh21:46:393
2730799118,2cyclictest31197-21expr20:25:153
2730799118,2cyclictest30717-21ssh23:10:143
2730799118,2cyclictest29496-21taskset19:13:203
2730799118,2cyclictest2879-21tune2fs19:25:173
2730799118,2cyclictest25915-21chrt23:51:063
2730799118,2cyclictest25343-21cut00:15:213
2730799118,2cyclictest25343-21cut00:15:213
2730799118,2cyclictest19332-21ssh23:46:383
2730799118,2cyclictest19320-21ssh22:16:293
2730799118,2cyclictest16077-21taskset19:51:543
2730799118,2cyclictest13563-21tail20:55:203
2730799118,2cyclictest1354-21grep20:30:163
2730799118,2cyclictest10696-21sed23:40:283
2730799118,2cyclictest10581-21ssh23:19:023
2730799118,2cyclictest10016-21taskset20:48:003
2730799115,2cyclictest224-21systemd-journal19:39:593
2730799113,7cyclictest6462-21ssh22:30:193
2730799112,7cyclictest17243-31sort00:10:063
2730799110,10cyclictest23321-21taskset21:56:423
2730799110,10cyclictest1474-21ssh00:22:563
2730499119,2cyclictest25992-21sh23:06:252
2730499119,1cyclictest77372sleep220:44:162
2730499119,1cyclictest5402-21grep19:30:202
2730499119,1cyclictest4928-21grep20:35:292
2730499119,1cyclictest30206-21grep20:20:292
2730499119,1cyclictest30206-21grep20:20:292
2730499119,1cyclictest28253-21tail19:10:262
2730499119,1cyclictest25339-21tail21:35:252
2730499119,1cyclictest20016-21ssh21:55:092
2730499119,1cyclictest12025-21wc20:50:272
2730499118,2cyclictest9231-21wc19:39:592
2730499118,2cyclictest3581-21grep19:25:272
2730499118,2cyclictest3029-21sh23:58:332
2730499118,2cyclictest3029-21sh23:58:332
2730499118,2cyclictest24335-21chrt20:10:192
2730499118,2cyclictest17786-21chrt21:05:132
2730499118,2cyclictest14447-21tr19:50:162
2730499112,1cyclictest9584-21aten2_r7power_v19:40:152
2730499110,11cyclictest0-21swapper/220:50:012
2730099118,2cyclictest8982-21ssh21:25:061
2730099118,2cyclictest8849-21expr20:45:191
2730099118,2cyclictest5880-21tr22:30:161
2730099118,2cyclictest32440-21cut19:20:161
2730099118,2cyclictest31249-21ssh00:20:211
2730099118,2cyclictest30089-21sort20:20:291
2730099118,2cyclictest30089-21sort20:20:291
2730099118,2cyclictest29623-21taskset23:31:371
2730099118,2cyclictest28961-21cat22:00:271
2730099118,2cyclictest27340-21cut21:15:151
2730099118,2cyclictest23217-21cut23:05:201
2730099118,2cyclictest22598-21grep20:05:221
2730099118,2cyclictest19923-21cron21:10:001
2730099118,2cyclictest18349-21ssh22:15:341
2730099118,2cyclictest17230-21sort00:35:271
2730099118,2cyclictest16293-21ssh22:36:571
2730099118,2cyclictest12339-21sshd20:52:171
2730099117,3cyclictest5442-21cat20:40:011
27300991110,1cyclictest0-21swapper/123:45:131
2729799119,2cyclictest20149-21cut21:55:140
2729799119,1cyclictest9868-21chrt20:46:070
2729799119,1cyclictest7267-21tr19:35:160
2729799119,1cyclictest32094-21cat23:55:280
2729799119,1cyclictest32094-21cat23:55:270
2729799119,1cyclictest28662-21ntpq19:10:290
2729799119,1cyclictest27840-21sed20:15:290
2729799119,1cyclictest19957-21ssh22:40:130
2729799119,1cyclictest1937-21ssh22:50:150
2729799119,1cyclictest182192sleep000:10:110
2729799119,1cyclictest16194-21ssh21:51:440
2729799118,2cyclictest9137-21ssh21:25:120
2729799118,2cyclictest6129-21ssh23:15:210
2729799118,2cyclictest4900-21tr19:30:180
2729799118,2cyclictest3781-21grep22:05:270
2729799118,2cyclictest24731-21cut00:15:170
2729799118,2cyclictest24731-21cut00:15:170
2729799118,2cyclictest15912-21cat21:00:160
2729799116,2cyclictest23724-21/usr/sbin/munin20:10:160
2729799110,11cyclictest30442-21ssh21:16:410
2729799110,10cyclictest3372-21ssh22:28:190
2729799110,10cyclictest27327-21ssh23:52:290
2730799108,1cyclictest6167-21cat19:30:313
2730799108,1cyclictest4120-21cat20:35:213
2730799108,1cyclictest23289-21taskset20:07:363
2730799108,1cyclictest19632-21taskset21:06:263
2730799108,1cyclictest1598-21grep19:20:293
2730799107,2cyclictest6278-21cut20:40:213
2730799106,3cyclictest27067-21ssh22:45:093
2730799100,9cyclictest29943-21swap20:20:283
2730799100,9cyclictest29943-21swap20:20:283
2730499109,1cyclictest31023-21sh23:10:162
2730499109,1cyclictest1759-21sh22:50:112
2730499108,1cyclictest31037-21cut20:25:152
2730499108,1cyclictest13289-21wc19:45:252
2730499107,2cyclictest994-21wget20:30:152
2730499107,2cyclictest30369-21expr19:15:172
2730499103,3cyclictest23491-21basename20:10:002
2730099108,2cyclictest26068-21ssh22:21:001
2730099108,1cyclictest4162sleep120:28:171
2730099108,1cyclictest4070-21chrt19:26:191
2730099108,1cyclictest28319-21expr19:10:271
2730099108,1cyclictest1700-21sh21:20:011
2730099107,2cyclictest5780-21ntpq19:30:261
2730099107,2cyclictest27909-21taskset20:15:491
2730099104,2cyclictest224-21systemd-journal19:15:001
2729799108,1cyclictest21925-21df20:05:160
2729799108,1cyclictest19418-21expr20:00:150
2729799108,1cyclictest18045-21cut21:05:180
2729799108,1cyclictest14812-21entropy19:50:180
2729799107,2cyclictest17361-21expr19:55:190
2729799103,5cyclictest0-21swapper/019:20:000
273009998,1cyclictest17210-21grep19:55:181
273009997,1cyclictest6467-21basename20:40:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional