You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 05:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 16, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14657210288,10sleep30-21swapper/319:09:083
1460829985,9sleep10-21swapper/119:08:291
1459728768,15sleep00-21swapper/019:08:200
1451728363,16sleep20-21swapper/219:07:192
244432250,0chrt94-21kswapd023:37:430
312682240,0sleep00-21swapper/023:42:280
14811992423,1cyclictest0-21swapper/022:58:560
14811992423,1cyclictest0-21swapper/022:58:560
14820992318,1cyclictest0-21swapper/322:07:523
1482099220,4cyclictest0-21swapper/321:50:483
14818992219,2cyclictest897-21kworker/2:200:10:182
1481199216,8cyclictest5486-21munin-run21:05:000
1482099191,2cyclictest0-21swapper/321:35:523
1481899190,1cyclictest0-21swapper/221:52:552
1481199194,7cyclictest21393-21munin-run22:05:000
14818991816,1cyclictest27111-21kworker/2:221:30:002
1481599180,1cyclictest0-21swapper/100:10:321
1481199186,6cyclictest15352-21munin-run23:55:000
1481199181,15cyclictest0-21swapper/021:50:140
1482099172,1cyclictest0-21swapper/323:46:323
1481899170,17cyclictest0-21swapper/222:08:152
14815991714,2cyclictest94-21kswapd022:10:141
14820991611,3cyclictest10695-21tail23:05:163
1481899161,1cyclictest10616-21fschecks_time23:05:172
1481899160,1cyclictest7861-21diskmemload22:58:512
1481899160,1cyclictest7861-21diskmemload22:58:502
1481899160,15cyclictest0-21swapper/222:20:542
1481599160,1cyclictest0-21swapper/100:08:581
14811991615,1cyclictest0-21swapper/000:16:480
1481199160,3cyclictest0-21swapper/020:30:190
1482099155,10cyclictest26437-21ssh00:02:033
14820991510,4cyclictest19980-21sh23:11:143
14818991515,0cyclictest0-21swapper/221:33:012
14818991512,3cyclictest7861-21diskmemload21:41:422
1481899150,15cyclictest0-21swapper/222:18:392
14815991512,2cyclictest0-21swapper/120:30:001
1481199152,7cyclictest466-21awk20:55:000
1482099142,11cyclictest0-21swapper/300:35:153
14820991412,1cyclictest7861-21diskmemload22:15:463
14820991410,3cyclictest31120-21cron23:20:003
14820991410,3cyclictest20496-21tr23:35:163
14820991410,3cyclictest11345-21tr00:15:233
14820991410,2cyclictest29230-21grep23:40:283
1482099140,13cyclictest0-21swapper/322:55:173
1482099140,13cyclictest0-21swapper/322:55:173
1482099140,0cyclictest0-21swapper/300:28:213
14818991413,1cyclictest5910-21kworker/2:123:11:182
14818991413,1cyclictest0-21swapper/221:10:042
1481899141,12cyclictest0-21swapper/222:43:292
1481899140,13cyclictest453-21grep20:55:002
1481899140,13cyclictest0-21swapper/221:49:052
1481599143,8cyclictest5847-21munin-run00:40:011
14815991411,2cyclictest29333-21ssh00:30:221
14815991411,2cyclictest27764-21head20:40:281
14815991410,3cyclictest4231-21cat19:55:001
14815991410,3cyclictest2012-21cut23:45:161
14815991410,3cyclictest16144-21grep22:45:271
14811991412,2cyclictest13-21ksoftirqd/021:38:010
1481199141,1cyclictest0-21swapper/022:45:140
1481199140,10cyclictest23230-21diskstats00:25:180
1482099139,3cyclictest19535-21sh19:20:013
1482099139,3cyclictest18716-21tr21:40:183
1482099139,3cyclictest12727-21cat23:30:003
1482099139,3cyclictest10790-21sed20:05:273
1482099139,2cyclictest22390-21grep19:25:193
1482099138,3cyclictest18366-21grep22:25:163
14820991310,2cyclictest7397-21grep22:40:153
14820991310,2cyclictest3742-21cron21:30:003
14820991310,2cyclictest29004-21grep00:30:193
14820991310,2cyclictest20926-21sed19:20:203
14820991310,2cyclictest18936-21cut20:25:163
14820991310,2cyclictest1210-21grep20:55:193
1482099130,1cyclictest0-21swapper/322:00:583
1482099130,13cyclictest0-21swapper/323:22:583
1481899139,3cyclictest26275-21sed19:30:262
1481899134,7cyclictest20505-21munin-node00:03:012
14818991312,1cyclictest61750irq/125-enp2s0-23:01:512
14818991310,2cyclictest29021-21sh21:25:012
14818991310,2cyclictest23020-21grep19:25:232
1481899130,13cyclictest0-21swapper/200:06:162
1481899130,12cyclictest0-21swapper/222:47:082
1481599139,3cyclictest10744-21sed20:05:281
1481599139,2cyclictest23197-21ls22:05:221
1481599138,3cyclictest15306-21tr19:10:181
1481599135,4cyclictest224-21systemd-journal23:15:001
1481599131,11cyclictest16341-21ssh23:09:391
1481599131,10cyclictest25-21ksoftirqd/121:05:151
14815991310,2cyclictest4985-21sed21:30:191
14815991310,2cyclictest25753-21cron20:39:591
14815991310,2cyclictest2456-21wc20:55:311
14815991310,2cyclictest1390-21ssh21:50:211
1481199139,3cyclictest15317-21grep19:10:180
14811991310,2cyclictest6868-21tr21:05:220
14811991310,2cyclictest4858-21grep21:30:180
14811991310,2cyclictest23758-21sed22:05:270
14811991310,2cyclictest17759-21grep23:55:280
14811991310,2cyclictest1648-21grep20:55:220
1481199130,9cyclictest0-21swapper/000:10:180
1481199130,13cyclictest0-21swapper/000:03:080
1481199130,12cyclictest6979-21tr20:00:140
1481199130,10cyclictest0-21swapper/022:25:010
326312120,3sleep00-21swapper/021:26:360
1482099129,2cyclictest9793-21cut23:50:183
1482099129,2cyclictest8146-21sort21:55:183
1482099129,2cyclictest30144-21tr00:05:163
1482099129,2cyclictest23182-21sed22:50:303
1482099129,2cyclictest15049-21sed23:30:273
1481899129,2cyclictest7766-21wc20:00:192
1481899129,2cyclictest24402-21grep20:35:202
1481899129,2cyclictest23040-21grep22:50:302
1481899129,2cyclictest1722-21grep20:55:222
1481899129,2cyclictest16369-21head21:15:202
1481899129,2cyclictest15672-21sort19:10:192
1481899129,2cyclictest15520-21grep20:15:292
1481899129,2cyclictest11958-21grep20:10:182
1481899129,1cyclictest5985-21cat23:25:012
1481899128,3cyclictest29832-21grep22:10:192
1481899126,2cyclictest224-21systemd-journal22:35:012
1481899124,7cyclictest17889-21runrttasks23:55:342
14818991212,0cyclictest0-21swapper/222:35:052
14818991211,1cyclictest0-21swapper/200:25:262
1481899120,1cyclictest0-21swapper/222:25:292
1481899120,12cyclictest0-21swapper/220:30:182
1481599129,2cyclictest8194-21head21:55:191
1481599129,2cyclictest5034-21awk22:15:211
1481599129,2cyclictest3691-21sort23:00:181
1481599129,2cyclictest3503-21grep21:00:151
1481599129,2cyclictest32485-21grep19:45:201
1481599129,2cyclictest32126-21wc20:50:241
1481599129,2cyclictest28661-21tr20:45:171
1481599129,2cyclictest23223-21grep22:50:291
1481599129,2cyclictest21378-21basename20:30:191
1481599129,2cyclictest19194-21head21:40:211
1481599129,2cyclictest18055-21grep19:15:201
1481599121,2cyclictest26743-21munin-run19:35:001
1481599120,1cyclictest0-21swapper/122:26:371
1481199129,2cyclictest8549-21ls21:55:210
1481199129,2cyclictest4463-21grep22:15:170
1481199129,2cyclictest3126-21sed00:35:290
1481199129,2cyclictest24730-21expr23:15:160
1481199129,2cyclictest21971-21cron19:25:000
1481199129,2cyclictest17933-21ssh23:10:180
1481199129,2cyclictest16193-21wc20:20:020
1481199129,2cyclictest11293-21grep22:20:170
1481199128,2cyclictest31362-21ssh21:49:230
1481199123,2cyclictest224-21systemd-journal22:35:000
1481199121,7cyclictest224-21systemd-journal23:01:450
1481199121,6cyclictest16756-21munin-run23:10:000
1481199121,6cyclictest0-21swapper/000:30:160
1482099118,2cyclictest7806-21taskset21:09:323
1482099118,2cyclictest5273-21cat19:55:223
1482099118,2cyclictest4737-21df21:30:173
1482099118,2cyclictest3824-21ssh23:00:193
1482099118,2cyclictest32427-21grep20:50:293
1482099118,2cyclictest27945-21ssh21:46:083
1482099118,2cyclictest27759-21tail19:35:193
1482099118,2cyclictest25471-21cut22:30:153
1482099118,2cyclictest24806-21ssh21:20:483
1482099118,2cyclictest2331-21ssh22:35:473
1482099118,2cyclictest2005-21cut19:50:143
1482099118,2cyclictest19893-21ssh00:22:063
1482099118,2cyclictest18945-21ssh23:56:423
1482099118,2cyclictest17124-21ssh22:24:233
1482099118,2cyclictest16006-21expr21:15:183
1482099118,2cyclictest13897-21sh22:45:013
1482099118,2cyclictest10439-21ssh21:10:273
1482099117,3cyclictest29875-21sed22:10:183
1481899119,1cyclictest28698-21head19:35:302
1481899118,2cyclictest6680-21uname21:05:202
1481899118,2cyclictest3823-21tail23:45:282
1481899118,2cyclictest3721-21chrt00:36:072
1481899118,2cyclictest3553-21sed19:50:262
1481899118,2cyclictest3414-21hostname21:00:142
1481899118,2cyclictest28462-21cut00:30:172
1481899118,2cyclictest25630-21cat23:15:232
1481899118,2cyclictest22079-21sed23:35:292
1481899118,2cyclictest16726-21cut00:20:162
1481899118,2cyclictest16022-21cron00:20:002
1481899118,2cyclictest15164-21expr22:00:192
1481899118,2cyclictest13346-21tr23:30:152
1481899118,2cyclictest10509-21sh23:50:232
1481899117,3cyclictest29213-21grep20:45:202
1481899116,4cyclictest6263-21ssh23:25:172
14818991110,1cyclictest20668-21if_enp0s31f619:20:182
1481899110,11cyclictest0-21swapper/223:40:122
1481599118,2cyclictest9849-21head23:50:181
1481599118,2cyclictest7855-21grep22:40:171
1481599118,2cyclictest32361-21grep23:20:181
1481599118,2cyclictest3221-21ssh22:36:391
1481599118,2cyclictest29762-21tr21:25:141
1481599118,2cyclictest26165-21tail21:45:191
1481599118,2cyclictest25510-21cat23:15:221
1481599118,2cyclictest22966-21head19:25:211
1481599118,2cyclictest20117-21grep19:20:151
1481599118,2cyclictest20066-21ssh23:35:101
1481599118,2cyclictest17834-21ssh23:55:321
1481599118,2cyclictest15902-21chrt20:18:381
1481599118,2cyclictest14895-21cat23:30:271
1481599118,2cyclictest1168-21ssh23:44:441
1481599118,2cyclictest10496-21ssh21:10:291
1481599117,3cyclictest7861-21diskmemload23:26:581
1481599117,2cyclictest11587-21ssh22:20:171
1481599110,4cyclictest0-21swapper/121:21:001
1481599110,11cyclictest0-21swapper/122:01:381
1481199119,1cyclictest8788-21tr21:10:140
1481199118,2cyclictest9150-21cut20:05:160
1481199118,2cyclictest7901-21grep22:40:170
1481199118,2cyclictest6482-21grep23:25:190
1481199118,2cyclictest5328-21ssh23:24:310
1481199118,2cyclictest3613-21ssh22:37:010
1481199118,2cyclictest29573-21ssh22:10:160
1481199118,2cyclictest29400-21apache_volume19:40:150
1481199118,2cyclictest25348-21head19:30:170
1481199118,2cyclictest23337-21needreboot19:25:240
1481199118,2cyclictest21604-21sh21:41:500
1481199118,2cyclictest21032-21ssh00:23:300
1481199118,2cyclictest19081-21ssh23:34:300
1481199118,2cyclictest16413-21wget20:20:160
1481199118,2cyclictest15930-21cat21:15:180
1481199115,5cyclictest1-21systemd21:24:210
1481199113,3cyclictest224-21systemd-journal19:36:040
1481199110,11cyclictest5448-21ssh23:47:010
1482099108,1cyclictest703-21systemctl19:45:253
1482099108,1cyclictest312522sleep319:41:563
1482099108,1cyclictest30233-21taskset20:46:163
1482099108,1cyclictest24495-21if_enp0s31f620:35:213
1482099107,2cyclictest7485-21tail20:00:163
1482099107,2cyclictest4914-21grep00:10:233
1482099107,2cyclictest27854-21chrt20:41:143
1482099107,2cyclictest24843-21grep19:30:153
1482099107,2cyclictest23064-21taskset20:31:023
1482099107,2cyclictest17967-21tail20:20:283
1482099107,2cyclictest15331-21tr19:10:183
1482099107,2cyclictest13914-21wget20:15:173
1482099107,2cyclictest12590-21tr20:10:233
1481899108,1cyclictest26134-21basename20:40:162
1481899108,1cyclictest20388-21grep20:25:282
1481899108,1cyclictest14192-21ssh21:36:372
1481899107,2cyclictest790-21cat19:45:292
1481899107,2cyclictest7182-21ssh21:55:032
1481899107,2cyclictest29855-21expr19:40:182
1481899100,9cyclictest17756-21cat19:15:192
1481899100,9cyclictest10009-21uname20:05:192
1481899100,4cyclictest224-21systemd-journal19:56:082
1481599108,1cyclictest4392-21aten2_r7power_p19:55:181
1481599108,1cyclictest16284-21ssh00:20:011
1481599108,1cyclictest16266-21taskset20:20:021
1481599108,1cyclictest15359-21ssh21:15:021
1481599108,1cyclictest11785-21cat20:10:171
1481599108,1cyclictest11622-21expr21:35:151
1481599107,2cyclictest9007-21cat20:05:001
1481599107,2cyclictest31326-21chrt19:42:501
1481599107,2cyclictest29908-21ssh22:55:281
1481599107,2cyclictest29908-21ssh22:55:281
1481599107,2cyclictest27726-21grep19:35:201
1481599107,2cyclictest23842-21sh00:00:191
1481599107,2cyclictest22531-21ssh00:25:111
1481599107,2cyclictest10913-21sed00:15:191
1481599100,9cyclictest25244-21ssh22:30:051
1481199108,1cyclictest4552-21cut19:55:180
1481199108,1cyclictest30401-21df_abs00:05:180
1481199108,1cyclictest28548-21basename20:45:180
1481199108,1cyclictest24060-21ssh22:51:130
1481199108,1cyclictest24027-21ls20:35:190
1481199108,1cyclictest1998-21hostname19:50:140
1481199108,1cyclictest18026-21awk19:15:200
1481199108,1cyclictest11533-21wget20:10:170
1481199107,2cyclictest19078-21expr20:25:180
148209992,2cyclictest224-21systemd-journal21:05:013
148189997,1cyclictest16724-21expr20:20:172
148119996,2cyclictest557-21grep19:45:240
148119990,9cyclictest26191-21wget20:40:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional