You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 11:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 02, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24624210184,13sleep10-21swapper/119:06:311
2477329682,9sleep00-21swapper/019:08:270
2469829066,8sleep20-21swapper/219:07:282
2455728067,9sleep30-21swapper/319:05:403
185122640,0sleep10-21swapper/121:29:471
2497699300,0cyclictest0-21swapper/021:19:230
2497899299,19cyclictest1160-21sendmail_mailtr22:55:221
2497899296,22cyclictest0-21swapper/121:31:351
24978992924,4cyclictest17564-21diskmemload00:25:131
24978992616,9cyclictest19780-21munin-run21:50:001
2498299250,25cyclictest0-21swapper/200:21:112
24978992410,1cyclictest0-21swapper/122:39:471
24978992410,13cyclictest0-21swapper/123:45:001
2498299230,23cyclictest0-21swapper/200:27:212
24978992320,2cyclictest17564-21diskmemload23:31:101
24978992219,2cyclictest0-21swapper/121:20:131
24978992215,6cyclictest17564-21diskmemload23:06:281
2498499210,20cyclictest0-21swapper/323:48:063
2498299210,0cyclictest141rcu_preempt21:16:012
2498499200,1cyclictest0-21swapper/323:27:233
2498499200,1cyclictest0-21swapper/321:41:533
2498299201,19cyclictest0-21swapper/222:48:092
2497899208,7cyclictest17564-21diskmemload21:50:461
24978992018,1cyclictest17564-21diskmemload00:18:591
24978992012,7cyclictest0-21swapper/123:41:411
2498499190,19cyclictest0-21swapper/321:15:503
24978991913,4cyclictest0-21swapper/100:20:161
24978991911,5cyclictest0-21swapper/100:07:541
24978991910,8cyclictest0-21swapper/121:35:591
2497899190,6cyclictest0-21swapper/100:37:141
24976991919,0cyclictest0-21swapper/022:32:500
24984991817,1cyclictest11144-21kworker/3:223:43:523
2498499180,17cyclictest7281-21kworker/3:122:30:323
24982991817,1cyclictest0-21swapper/223:40:132
24982991812,5cyclictest29673-21apache223:25:442
2498299180,18cyclictest0-21swapper/221:57:002
2498299180,17cyclictest0-21swapper/223:57:272
2497899189,8cyclictest0-21swapper/122:16:131
2497899188,9cyclictest17564-21diskmemload22:12:421
2497899186,11cyclictest0-21swapper/122:54:051
24978991812,5cyclictest537-21snmpd23:27:001
24978991810,7cyclictest28377-21mailstats23:50:231
2497699188,2cyclictest141rcu_preempt23:30:130
2497699186,6cyclictest18790-21munin-run21:29:590
2497699184,1cyclictest141rcu_preempt23:23:010
2498499172,11cyclictest21279-21ssh22:29:243
24984991716,1cyclictest7281-21kworker/3:122:23:423
2498499170,2cyclictest0-21swapper/323:57:283
2498299170,1cyclictest141rcu_preempt21:47:032
24978991715,2cyclictest17564-21diskmemload00:10:171
24978991715,1cyclictest17564-21diskmemload22:01:541
24978991713,3cyclictest19584-21mailstats21:10:211
24978991713,3cyclictest0-21swapper/121:19:041
24978991713,1cyclictest23457-21sed22:30:181
24978991711,5cyclictest17564-21diskmemload00:04:281
2497899171,13cyclictest0-21swapper/123:00:051
2498499166,8cyclictest475-21in:imuxsock21:29:593
24984991615,1cyclictest0-21swapper/323:15:453
24984991615,1cyclictest0-21swapper/322:57:213
24982991615,1cyclictest0-21swapper/221:12:082
2498299160,16cyclictest0-21swapper/222:55:532
2498299160,16cyclictest0-21swapper/222:13:322
2497899169,7cyclictest0-21swapper/122:09:451
2497899167,4cyclictest0-21swapper/123:11:211
24978991615,1cyclictest0-21swapper/121:55:151
24978991613,2cyclictest0-21swapper/122:25:141
24978991611,4cyclictest0-21swapper/122:21:591
2497699164,6cyclictest11858-21idleruntime-cro19:50:000
24984991513,1cyclictest0-21swapper/300:20:113
2498499150,1cyclictest0-21swapper/300:27:203
2498499150,15cyclictest0-21swapper/323:02:083
24982991515,0cyclictest0-21swapper/223:22:172
24982991514,1cyclictest311rcuc/222:25:162
2498299151,1cyclictest141rcu_preempt23:17:032
24982991511,2cyclictest2889-21wc23:35:202
2498299150,2cyclictest2215-21ssh21:20:012
2497899158,5cyclictest17564-21diskmemload23:22:301
24978991512,2cyclictest17564-21diskmemload22:45:151
24978991511,3cyclictest0-21swapper/121:40:401
24978991510,5cyclictest0-21swapper/123:58:441
2497699151,8cyclictest6422-21munin-run22:40:000
2497699150,2cyclictest0-21swapper/023:11:200
24984991410,3cyclictest28290-21sh23:50:223
2498499140,13cyclictest16576-21cron20:00:003
24982991411,2cyclictest21347-21cron20:10:002
24982991410,3cyclictest7500-21wc20:45:272
24982991410,3cyclictest26431-21sed19:10:232
2498299140,14cyclictest0-21swapper/223:12:122
2498299140,14cyclictest0-21swapper/222:17:092
2497899142,11cyclictest18974-21cron20:05:001
24978991410,3cyclictest15722-21tr21:05:141
2497699141,7cyclictest28904-21munin-run00:30:000
2497699140,2cyclictest0-21swapper/022:04:410
93412130,1sleep00-21swapper/023:39:470
2498499139,3cyclictest404-21tr19:25:153
2498499138,3cyclictest27255-21cat21:35:003
2498499134,5cyclictest225-21systemd-journal22:40:003
24984991311,1cyclictest0-21swapper/322:15:153
24984991310,2cyclictest26702-21tail23:30:173
24984991310,2cyclictest20356-21head00:05:173
2498499130,1cyclictest27936-21ssh22:52:153
2498499130,11cyclictest0-21swapper/323:14:493
2498299139,3cyclictest3788-21grep20:40:152
2498299139,3cyclictest28115-21cron21:55:002
2498299139,2cyclictest0-21swapper/200:35:142
24982991311,1cyclictest32-21ksoftirqd/222:01:142
24982991310,2cyclictest9782-21wc20:50:252
24982991310,2cyclictest32424-21wc22:35:222
24982991310,2cyclictest32-21ksoftirqd/200:17:272
24982991310,2cyclictest30530-21tr19:20:152
24982991310,2cyclictest28501-21cron20:25:012
24982991310,2cyclictest27806-21grep23:30:262
24982991310,2cyclictest21204-21grep21:30:272
24982991310,2cyclictest17801-21wc23:05:222
2497899139,3cyclictest0-21swapper/100:34:071
2497899139,3cyclictest0-21swapper/100:34:061
2497899131,1cyclictest2086-21ssh23:15:201
24978991311,1cyclictest0-21swapper/119:15:141
24978991310,2cyclictest7598-21grep19:40:141
24978991310,2cyclictest7061-21cron19:39:591
24978991310,2cyclictest3371-21grep19:30:181
24978991310,2cyclictest2554-21grep23:35:161
2497699139,3cyclictest21505-21grep00:05:270
2497699134,5cyclictest0-21swapper/021:10:140
24976991310,2cyclictest8803-21sed23:00:150
24976991310,2cyclictest4749-21ls20:40:230
24976991310,2cyclictest3319-21grep19:30:180
24976991310,2cyclictest3317-21cat20:40:000
24976991310,2cyclictest16715-21cat20:00:010
24976991310,2cyclictest13182-21needreboot21:45:210
24976991310,2cyclictest13099-21cron00:40:000
24976991310,2cyclictest11187-21chrt22:42:110
24976991310,2cyclictest11187-21chrt22:42:110
2498499129,2cyclictest6654-21sed19:35:263
2498499129,2cyclictest3822-21tr19:30:233
2498499129,2cyclictest31352-21grep00:30:273
2498499129,2cyclictest31352-21grep00:30:273
2498499129,2cyclictest29937-21grep20:25:213
2498499129,2cyclictest28146-21basename21:55:003
2498499129,2cyclictest24277-21grep20:15:153
2498499129,2cyclictest18997-21basename20:05:003
2498499129,2cyclictest18834-21grep21:10:163
2498499129,2cyclictest16619-21tr23:05:143
2498499129,2cyclictest13559-21grep21:00:153
2498499129,2cyclictest12889-21head19:50:173
2498499128,3cyclictest1910-21head20:35:163
2498499126,5cyclictest22677-21ssh22:10:223
2498499124,4cyclictest225-21systemd-journal20:50:013
2498499120,2cyclictest141rcu_preempt23:22:533
2498299129,2cyclictest880-21latency_hist20:35:002
2498299129,2cyclictest8736-21grep23:00:152
2498299129,2cyclictest8115-21grep19:40:162
2498299129,2cyclictest3674-21wc00:15:002
2498299129,2cyclictest29934-21grep20:25:212
2498299129,2cyclictest27677-21sed23:50:172
2498299129,2cyclictest25325-21systemd-cgroups00:08:142
2498299129,2cyclictest24488-21grep22:50:172
2498299129,2cyclictest13668-21sort21:00:172
2498299129,2cyclictest12798-21cut19:50:162
2498299129,2cyclictest12400-21wc21:25:212
24982991212,0cyclictest0-21swapper/222:22:122
24982991211,1cyclictest0-21swapper/222:42:082
24982991211,1cyclictest0-21swapper/222:42:082
2498299120,1cyclictest0-21swapper/221:37:222
2498299120,12cyclictest0-21swapper/222:07:312
2497899129,2cyclictest31931-21grep20:30:181
2497899129,2cyclictest22721-21sed20:10:211
2497899128,3cyclictest24778-21grep20:15:171
2497899127,4cyclictest0-21swapper/121:00:271
2497899127,2cyclictest0-21swapper/122:43:271
2497899127,2cyclictest0-21swapper/122:43:271
2497899126,5cyclictest0-21swapper/120:35:151
2497899122,3cyclictest225-21systemd-journal20:30:001
2497699129,2cyclictest7690-21wc23:00:010
2497699129,2cyclictest6646-21grep20:45:180
2497699129,2cyclictest6089-21cat00:15:270
2497699129,2cyclictest3166-21df21:20:150
2497699129,2cyclictest30776-21ls00:30:220
2497699129,2cyclictest30776-21ls00:30:220
2497699129,2cyclictest22348-21tail20:10:180
2497699129,2cyclictest20482-21grep23:45:270
2497699129,2cyclictest13472-21sed00:20:170
2497699128,3cyclictest19053-21wc20:05:000
2497699127,2cyclictest20245-21/usr/sbin/munin21:50:160
2497699125,6cyclictest14322-21ssh22:05:210
24976991210,1cyclictest11255-21grep00:00:140
2497699120,6cyclictest26141-21awk20:20:000
2497699120,11cyclictest0-21swapper/023:56:520
2498499118,2cyclictest9687-21chrt19:45:033
2498499118,2cyclictest7541-21ssh00:35:373
2498499118,2cyclictest7368-21hostname19:40:133
2498499118,2cyclictest7084-21aten2_r7power_v22:40:153
2498499118,2cyclictest7084-21aten2_r7power_v22:40:143
2498499118,2cyclictest5445-21grep22:00:173
2498499118,2cyclictest4031-21cut21:20:223
2498499118,2cyclictest30885-21ssh00:11:023
2498499118,2cyclictest28494-21ssh21:55:073
2498499118,2cyclictest28305-21grep21:35:163
2498499118,2cyclictest26653-21grep20:20:153
2498499118,2cyclictest17202-21ssh22:45:293
2498499118,2cyclictest16087-21grep21:05:163
2498499118,2cyclictest1554-21ssh23:35:023
2498499118,2cyclictest14374-21ssh00:01:223
2498499118,2cyclictest14013-21ssh21:45:313
2498499118,2cyclictest12760-21ssh22:05:033
2498499117,3cyclictest5705-21grep00:15:233
2498499112,8cyclictest29768-21sendmail19:20:003
2498299119,1cyclictest29370-1kworker/u9:200:30:122
2498299119,1cyclictest29370-1kworker/u9:200:30:122
2498299118,2cyclictest9833-21tr19:45:132
2498299118,2cyclictest4694-21dump-pmu-power19:35:002
2498299118,2cyclictest28210-21cut19:15:172
2498299118,2cyclictest25137-21ssh22:30:512
2498299118,2cyclictest24060-21cut20:15:142
2498299118,2cyclictest23677-21ssh23:47:552
2498299118,2cyclictest17587-21head20:00:172
2498299118,2cyclictest17156-21head21:05:262
2498299118,2cyclictest12322-21tail20:55:272
2498299118,2cyclictest11481-21cut00:00:152
2497899119,1cyclictest12074-21acpi19:50:111
2497899119,1cyclictest10008-21grep20:50:261
2497899118,2cyclictest536-21expr19:25:141
2497899118,2cyclictest25397-21chrt19:10:151
2497899118,2cyclictest14739-21tr19:55:171
2497899118,2cyclictest12805-21basename21:00:001
2497699118,2cyclictest9650-21wc19:45:000
2497699118,2cyclictest5629-21ssh21:40:270
2497699118,2cyclictest415-21grep19:25:140
2497699118,2cyclictest3542-21cron00:15:000
2497699118,2cyclictest30140-21dpkg19:20:010
2497699118,2cyclictest29456-21run-parts19:17:010
2497699118,2cyclictest27180-21head20:20:180
2497699118,2cyclictest26923-21ssh23:50:140
2497699118,2cyclictest26407-21ls19:10:230
2497699118,2cyclictest24456-21tr22:50:170
2497699118,2cyclictest24040-21sh23:29:010
2497699118,2cyclictest22592-21uname22:10:210
2497699118,2cyclictest20858-21grep20:05:250
2497699118,2cyclictest20563-21tr21:30:200
2497699118,2cyclictest19549-21ssh23:06:260
2497699118,2cyclictest14139-21cat21:00:200
2497699118,2cyclictest1199-21ssh21:57:450
2497699118,2cyclictest1185-21expr23:15:150
2497699118,2cyclictest11667-21ssh22:23:230
2497699117,3cyclictest13758-21grep19:50:260
2497699117,3cyclictest11851-21ssh23:40:250
2497699117,1cyclictest31482-21ssh21:36:510
2497699112,4cyclictest225-21systemd-journal22:25:000
2498499108,1cyclictest9988-21awk20:50:263
2498499108,1cyclictest321642sleep319:22:313
2498499108,1cyclictest271652sleep319:13:103
2498499108,1cyclictest212452sleep320:08:443
2498499107,2cyclictest5180-21grep20:40:273
2498499107,2cyclictest22110-21df20:10:163
2498499107,2cyclictest11378-21grep20:55:173
2498499100,9cyclictest32061-21mii-tool20:30:193
2498299108,1cyclictest6954-21taskset19:38:412
2498299108,1cyclictest4077-21ssh21:40:142
2498299108,1cyclictest144882sleep219:55:162
2498299107,2cyclictest22236-21grep20:10:182
2498299107,2cyclictest2206-21taskset19:28:392
2498299107,2cyclictest1150-21chrt20:35:112
2497899108,1cyclictest10151-21expr19:45:141
2497899108,1cyclictest0-21swapper/120:05:121
2497899107,2cyclictest0-21swapper/119:20:131
2497899100,9cyclictest6411-21df20:45:161
2497899100,7cyclictest4955-21/usr/sbin/munin20:40:261
2497699108,1cyclictest308882sleep020:29:280
2497699108,1cyclictest29604-21ssh22:15:090
2497699108,1cyclictest17699-21ssh22:45:460
2497699107,2cyclictest8304-21dpkg20:50:120
2497699107,2cyclictest31965-21head20:30:180
2497699107,2cyclictest17149-21sed21:05:260
2497699105,2cyclictest225-21systemd-journal19:39:590
249789997,1cyclictest26943-21df_abs20:20:161
249769997,1cyclictest12694-21chrt20:59:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional