You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 11:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Nov 18, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18514210289,9sleep30-21swapper/319:07:223
1850329686,6sleep20-21swapper/219:07:132
1865629474,16sleep10-21swapper/119:09:081
1848128466,13sleep00-21swapper/019:06:540
87032710,0sleep10-21swapper/121:30:201
18816992827,1cyclictest0-21swapper/223:43:072
1881899250,1cyclictest0-21swapper/322:19:533
1881899240,2cyclictest0-21swapper/321:48:083
1881899240,1cyclictest0-21swapper/322:35:043
1881899230,3cyclictest0-21swapper/322:06:073
1881899230,1cyclictest0-21swapper/321:19:173
1881699230,23cyclictest0-21swapper/221:36:412
204592200,0sleep00-21swapper/021:15:280
18816992019,1cyclictest4268-21kworker/2:222:03:332
18811992017,2cyclictest25925-21ssh22:05:150
1881199200,19cyclictest0-21swapper/022:18:570
1881199200,11cyclictest0-21swapper/000:00:150
1881899190,1cyclictest0-21swapper/323:08:003
1881899190,0cyclictest0-21swapper/323:15:143
1881699190,18cyclictest0-21swapper/219:40:172
1881899180,3cyclictest0-21swapper/322:00:563
1881899180,1cyclictest0-21swapper/323:22:583
1881699180,18cyclictest0-21swapper/221:50:042
1881699180,17cyclictest32257-21latency_hist20:45:002
1881199186,6cyclictest28328-21munin-run19:30:010
18811991818,0cyclictest0-21swapper/022:00:230
1881899172,2cyclictest4227-21apache_accesses21:50:143
18818991715,1cyclictest0-21swapper/321:28:523
1881699172,1cyclictest31198-21ssh00:25:462
18815991717,0cyclictest0-21swapper/122:17:421
1881599171,1cyclictest141rcu_preempt23:45:321
1881199175,7cyclictest15383-21idleruntime-cro20:10:000
18818991613,1cyclictest0-21swapper/300:20:253
1881899160,1cyclictest0-21swapper/321:33:053
18816991615,1cyclictest0-21swapper/223:28:032
1881699160,1cyclictest0-21swapper/223:50:492
1881699160,1cyclictest0-21swapper/223:15:442
1881699160,1cyclictest0-21swapper/200:09:222
1881699160,16cyclictest0-21swapper/221:20:412
1881599160,16cyclictest0-21swapper/100:00:261
1881199160,3cyclictest0-21swapper/000:08:370
18818991514,1cyclictest0-21swapper/321:40:283
1881899150,1cyclictest0-21swapper/322:31:223
1881699154,7cyclictest0-21swapper/223:35:132
18816991512,3cyclictest37850irq/131-snd_hda23:08:592
18816991510,4cyclictest7903-21head23:45:172
1881699150,15cyclictest0-21swapper/222:32:372
1881699150,0cyclictest0-21swapper/221:12:112
1881599156,5cyclictest15858-21apache223:38:441
18815991512,2cyclictest500-21in:imuxsock00:15:001
18815991510,3cyclictest29331-21cron22:30:001
1881199152,12cyclictest2507-21awk21:25:330
1881899146,3cyclictest224-21systemd-journal23:02:213
18818991411,2cyclictest23266-21wc23:10:273
1881899140,0cyclictest0-21swapper/323:47:213
1881899140,0cyclictest0-21swapper/321:39:513
18816991411,2cyclictest3557-21df00:30:162
1881599141,1cyclictest141rcu_preempt22:54:461
18815991411,2cyclictest3524-21sed20:50:231
18815991411,2cyclictest25-21ksoftirqd/122:10:001
1881599140,14cyclictest0-21swapper/122:22:571
1881199141,11cyclictest0-21swapper/023:45:140
18811991410,3cyclictest9334-21cat21:30:270
18811991410,3cyclictest21684-21grep21:40:000
18811991410,3cyclictest19289-21grep19:10:170
1881899136,2cyclictest224-21systemd-journal21:15:003
1881899134,4cyclictest224-21systemd-journal20:20:003
18818991310,2cyclictest938-21ls19:40:013
18818991310,2cyclictest3998-21grep20:50:283
18818991310,2cyclictest23691-21cat22:25:213
18818991310,2cyclictest22103-21cut23:55:183
1881899130,1cyclictest0-21swapper/300:31:383
1881899130,12cyclictest30283-21grep20:40:173
1881699139,3cyclictest4005-21ssh21:26:572
1881699138,4cyclictest28125-21munin-plugin-st23:15:012
1881699138,4cyclictest17242-21ssh22:43:512
1881699138,2cyclictest16236-21df20:10:172
1881699137,5cyclictest31791-21grep19:35:182
1881699134,2cyclictest0-21swapper/221:00:152
1881699133,5cyclictest0-21swapper/222:00:002
18816991310,2cyclictest19845-21head22:45:222
1881599139,3cyclictest25646-21chrt23:57:421
1881599139,3cyclictest1646-21grep23:40:301
1881599139,3cyclictest14477-21tr20:05:241
1881599139,3cyclictest0-21swapper/122:46:481
1881599135,7cyclictest0-21swapper/121:55:151
1881599134,2cyclictest224-21systemd-journal21:05:001
18815991311,2cyclictest25-21ksoftirqd/123:01:301
18815991310,2cyclictest4845-21cut20:55:191
18815991310,2cyclictest3597-21tr23:20:161
18815991310,2cyclictest29807-21tail00:25:181
18815991310,2cyclictest29707-21cron20:40:001
18815991310,2cyclictest27340-21turbostat.cron20:34:591
18815991310,2cyclictest23909-21wc20:25:221
18815991310,2cyclictest18806-21cut22:00:151
1881599130,12cyclictest7198-21cron21:30:011
1881199139,3cyclictest31917-21cut19:35:180
1881199139,3cyclictest31428-21chrt00:26:020
1881199139,3cyclictest12667-21taskset21:55:210
1881199137,4cyclictest1409-21ssh22:10:180
1881199133,3cyclictest9234-21perf21:05:000
1881199131,6cyclictest28839-21munin-run21:45:000
18811991310,2cyclictest5421-21tail20:55:210
18811991310,2cyclictest31537-21wc22:30:280
18811991310,2cyclictest2625-21wc19:40:290
18811991310,2cyclictest24462-21sed20:25:280
18811991310,2cyclictest1573-21sed20:45:280
18811991310,2cyclictest12430-21cut22:40:210
1881199130,12cyclictest17791-21sed00:15:280
1881899129,2cyclictest32410-21cut23:40:183
1881899129,2cyclictest29814-21head00:25:193
1881899129,2cyclictest23590-21head20:25:193
1881899129,2cyclictest17687-21basename23:30:143
1881899129,2cyclictest1251-21grep22:55:193
1881899129,2cyclictest12271-21sed23:25:283
18818991211,1cyclictest0-21swapper/323:36:173
1881899120,12cyclictest0-21swapper/322:45:263
1881899120,11cyclictest0-21swapper/321:57:573
1881699129,2cyclictest9785-21taskset21:30:372
1881699129,2cyclictest9728-21wget00:10:162
1881699129,2cyclictest8353-21tail23:00:192
1881699129,2cyclictest29880-21cat20:40:002
1881699129,2cyclictest28029-21sendmail23:59:592
1881699129,2cyclictest27366-21cut22:50:232
1881699129,2cyclictest25538-21wc20:30:152
1881699129,2cyclictest23460-21cut00:20:192
1881699129,2cyclictest22771-21wc20:24:592
1881699129,2cyclictest10976-21wc00:35:252
1881699127,4cyclictest7248-21ntpq19:50:232
18816991212,0cyclictest0-21swapper/222:37:522
18816991211,1cyclictest0-21swapper/219:30:142
1881699120,8cyclictest0-21swapper/222:56:302
1881699120,4cyclictest224-21systemd-journal21:20:012
1881699120,11cyclictest0-21swapper/222:20:132
1881599129,2cyclictest6959-21sh21:51:141
1881599129,2cyclictest6351-21tr19:50:171
1881599129,2cyclictest3358-21tr00:05:141
1881599129,2cyclictest30243-21tail19:30:271
1881599129,2cyclictest29954-21ssh23:15:241
1881599129,2cyclictest29334-21awk21:45:131
1881599129,2cyclictest27906-21head19:25:271
1881599129,2cyclictest23591-21wc19:20:011
1881599129,2cyclictest21689-21sed20:20:221
1881599129,2cyclictest20626-21head21:15:291
1881599129,2cyclictest16266-21tr00:15:161
1881599129,2cyclictest15317-21grep23:05:211
1881599129,2cyclictest12890-21grep21:10:211
1881599129,2cyclictest12521-21grep22:40:201
1881599128,3cyclictest4037-21wc00:30:201
1881599128,3cyclictest1200-21grep22:10:161
18815991210,1cyclictest25-21ksoftirqd/121:24:421
1881199129,2cyclictest4628-21grep22:35:160
1881199129,2cyclictest3061-21grep00:30:150
1881199129,2cyclictest29703-21cron20:40:000
1881199129,2cyclictest21602-21cut23:10:140
1881199129,2cyclictest16463-21grep23:05:300
1881199129,2cyclictest16451-21grep20:10:190
1881199129,2cyclictest15178-21sleep00:40:000
1881199128,3cyclictest20507-21grep20:20:140
1881199128,2cyclictest12358-21wc20:00:260
1881199124,7cyclictest15845-21apache221:53:520
1881199122,8cyclictest18283-21cron21:15:000
1881199120,12cyclictest0-21swapper/023:50:490
1881199120,11cyclictest0-21swapper/023:19:530
1881899119,1cyclictest29305-21grep00:00:183
1881899118,2cyclictest903-21grep22:10:143
1881899118,2cyclictest8865-21chrt21:00:453
1881899118,2cyclictest5171-21expr20:55:183
1881899118,2cyclictest3770-21tr19:45:173
1881899118,2cyclictest3383-21tr00:05:153
1881899118,2cyclictest31336-21ssh22:53:523
1881899118,2cyclictest26283-21cat20:30:203
1881899118,2cyclictest25707-21taskset19:22:153
1881899118,2cyclictest25649-21ssh21:20:103
1881899118,2cyclictest21695-21grep19:15:153
1881899118,2cyclictest18484-21ssh00:16:083
1881899118,2cyclictest17162-21ssh22:43:463
1881899118,2cyclictest15170-21rm22:20:083
1881899118,2cyclictest14291-21basename23:50:173
1881899118,2cyclictest10698-21grep21:05:223
1881899118,2cyclictest10034-21cat00:35:193
18818991110,1cyclictest0-21swapper/300:10:133
1881699119,1cyclictest2564-21grep20:50:162
1881699118,2cyclictest8802-21expr19:55:182
1881699118,2cyclictest6803-21cron21:00:002
1881699118,2cyclictest647-21grep20:45:192
1881699118,2cyclictest31417-21ssh22:08:512
1881699118,2cyclictest29306-21grep00:00:182
1881699118,2cyclictest22860-21cut22:25:172
1881699118,2cyclictest22383-21cat19:15:202
1881699118,2cyclictest21427-21ssh23:32:102
1881699118,2cyclictest19156-21needreboot20:15:222
1881699118,2cyclictest18606-21ssh00:16:162
1881699118,2cyclictest12632-21chrt20:00:362
1881699118,2cyclictest10958-21cat21:05:272
1881699118,1cyclictest5380-21sh23:20:362
1881699118,1cyclictest1410-21fschecks_time22:10:182
1881699114,5cyclictest29281-21apt-get21:45:132
1881699110,2cyclictest0-21swapper/220:06:452
1881699110,1cyclictest0-21swapper/221:40:272
1881599118,2cyclictest7744-21snmpd23:12:231
1881599118,2cyclictest3975-21tr19:45:181
1881599118,2cyclictest31111-21cut20:40:221
1881599118,2cyclictest19173-21ssh23:30:231
1881599118,2cyclictest17155-21ssh21:35:451
1881599118,2cyclictest15451-21ssh23:28:241
1881599118,2cyclictest11558-21cut20:00:171
1881599118,2cyclictest10177-21grep19:55:301
1881599118,2cyclictest10007-21expr00:35:181
1881599117,3cyclictest221irq_work/123:52:471
1881599117,3cyclictest19767-21grep20:15:281
1881599117,2cyclictest22456-21wc21:40:161
1881599115,2cyclictest141rcu_preempt22:32:141
1881599114,3cyclictest224-21systemd-journal19:22:341
1881599111,8cyclictest5496-21ssh22:35:211
1881199119,1cyclictest9549-21cut21:05:150
1881199118,2cyclictest9589-21head19:55:240
1881199118,2cyclictest5755-21ssh22:58:480
1881199118,2cyclictest30714-21grep20:40:180
1881199118,2cyclictest29265-21apache_processe21:45:110
1881199118,2cyclictest28827-21tr19:30:160
1881199118,2cyclictest28497-21sh22:51:000
1881199118,2cyclictest28237-21ssh21:20:580
1881199118,2cyclictest27626-21ssh22:28:240
1881199118,2cyclictest26845-21sed20:30:260
1881199118,2cyclictest24578-21cut19:20:200
1881199118,2cyclictest23603-21ls00:20:200
1881199118,2cyclictest21065-21ssh22:45:370
1881199118,2cyclictest18888-21grep23:30:200
1881199118,2cyclictest18665-21sort20:15:180
1881199118,2cyclictest13186-21ssh00:12:240
1881199118,2cyclictest1275-21grep23:40:260
1881199118,2cyclictest12750-21ssh23:03:420
1881199117,3cyclictest4514-21grep19:45:210
1881199116,4cyclictest25255-21cstates23:35:160
1881199110,3cyclictest0-21swapper/022:21:530
1881899108,1cyclictest6694-21forks19:50:173
1881899108,1cyclictest29475-21chrt20:37:093
1881899108,1cyclictest26428-21grep19:25:153
1881899108,1cyclictest19449-21expr19:10:163
1881899107,2cyclictest32607-21tr20:45:153
1881899107,2cyclictest30917-21cat19:35:003
1881899107,2cyclictest2299-21ntpq19:40:233
1881899107,2cyclictest12568-21sed20:00:273
1881899100,9cyclictest8756-21cut19:55:183
1881899100,9cyclictest19834-21taskset20:15:443
1881699108,1cyclictest5420-21awk19:45:302
1881699108,1cyclictest27567-21run-parts19:25:232
1881699108,1cyclictest24209-21which20:25:252
1881699107,2cyclictest19313-21grep19:10:172
1881699100,9cyclictest10254-21ssh22:15:442
1881599108,1cyclictest1102-21ntp_states20:45:231
1881599108,1cyclictest10848-21tr21:05:241
1881599107,2cyclictest2345-21sed22:55:291
1881599107,2cyclictest23339-21basename00:20:181
1881599107,2cyclictest21311-21ls19:15:011
1881599105,3cyclictest856-21sendmail19:40:011
1881599100,9cyclictest16324-21grep20:10:181
1881199108,1cyclictest6087-21bc19:50:130
1881199108,1cyclictest3287-21rm23:20:140
1881199108,1cyclictest2416-21cut20:50:150
1881199108,1cyclictest232302sleep019:16:160
1881199106,2cyclictest23245-21sed23:55:270
1881199100,9cyclictest10217-21ssh23:25:060
188189998,1cyclictest13853-21fschecks_count20:05:193
188189997,1cyclictest16016-21expr20:10:173
188169997,1cyclictest24660-21cat19:20:202
188159991,4cyclictest224-21systemd-journal19:41:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional