You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-06 - 14:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 06, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
798729266,9sleep00-21swapper/019:08:100
811328774,9sleep20-21swapper/219:09:482
793728157,9sleep30-21swapper/319:07:323
32662800,0sleep30-21swapper/300:24:483
803027659,13sleep10-21swapper/119:08:441
239812590,0sleep30-21swapper/322:20:543
119522510,0sleep10-21swapper/123:12:171
820799290,29cyclictest0-21swapper/223:59:002
820799248,2cyclictest141rcu_preempt23:01:212
820799230,23cyclictest0-21swapper/221:21:542
821099220,20cyclictest0-21swapper/323:51:523
820799221,16cyclictest0-21swapper/219:40:122
820699226,15cyclictest28444-21hddtemp_smartct21:45:171
820599217,2cyclictest8202-21cyclictest23:29:240
8210992020,0cyclictest0-21swapper/323:15:383
8205992020,0cyclictest0-21swapper/023:43:450
8205992020,0cyclictest0-21swapper/023:22:460
821099197,10cyclictest0-21swapper/320:30:143
8210991917,1cyclictest0-21swapper/322:02:043
820799190,18cyclictest0-21swapper/220:25:142
8206991919,0cyclictest0-21swapper/122:34:311
8210991817,1cyclictest0-21swapper/321:33:303
8210991815,2cyclictest475-21in:imuxsock21:11:033
821099180,18cyclictest0-21swapper/323:29:273
820699180,2cyclictest141rcu_preempt22:18:491
821099172,11cyclictest13535-21perf22:35:013
8210991716,0cyclictest0-21swapper/323:05:143
821099171,5cyclictest0-21swapper/321:35:133
820799172,14cyclictest3691-21aten2_r7power_p21:50:152
8207991715,1cyclictest1219-21diskmemload00:25:132
8207991711,2cyclictest30169-21latency_hist22:45:012
8206991714,2cyclictest0-21swapper/123:36:471
820599170,0cyclictest0-21swapper/023:38:220
821099164,8cyclictest1-21systemd00:39:593
821099160,15cyclictest0-21swapper/321:56:513
820799168,1cyclictest0-21swapper/222:23:372
8206991614,1cyclictest9750-21ssh00:09:241
8206991614,1cyclictest9750-21ssh00:09:231
8206991613,2cyclictest0-21swapper/123:40:101
820699161,2cyclictest17660-21taskset22:55:591
820699160,11cyclictest0-21swapper/123:09:551
820599166,5cyclictest27433-21apt-get00:20:000
820599164,6cyclictest0-21swapper/000:25:140
820599163,7cyclictest23776-21sendmail-msp23:19:590
8205991613,3cyclictest13-21ksoftirqd/023:56:540
8210991514,1cyclictest0-21swapper/323:57:383
8210991513,1cyclictest3805-21cat21:50:163
8207991514,1cyclictest0-21swapper/223:49:202
820799150,15cyclictest0-21swapper/222:03:052
820699158,6cyclictest22367-21sed22:00:301
820699150,2cyclictest0-21swapper/100:01:091
8205991513,2cyclictest13-21ksoftirqd/021:58:420
8205991512,2cyclictest7622-21cut22:50:170
820599151,13cyclictest27044-21cron21:44:590
820599150,2cyclictest0-21swapper/023:46:320
820599150,13cyclictest0-21swapper/000:20:380
8210991413,1cyclictest0-21swapper/300:18:393
821099141,12cyclictest0-21swapper/323:02:233
8210991410,3cyclictest5033-21cut22:10:153
821099140,14cyclictest0-21swapper/323:13:073
821099140,0cyclictest0-21swapper/300:02:133
820799145,9cyclictest49-21kcompactd000:13:232
820799142,5cyclictest1142-21runrttasks21:55:392
8207991413,1cyclictest0-21swapper/221:49:442
8207991410,3cyclictest584-21wget20:00:122
8207991410,3cyclictest10845-21wc19:15:012
820699144,8cyclictest32728-21apt-get23:25:161
8206991412,2cyclictest0-21swapper/122:41:021
820699140,10cyclictest0-21swapper/100:10:051
8205991410,3cyclictest9910-21tr23:50:150
8205991410,3cyclictest14039-21grep19:20:190
820599140,0cyclictest0-21swapper/022:34:270
821099139,3cyclictest24314-21aten2_r7power_c23:20:143
821099135,4cyclictest225-21systemd-journal19:20:003
8210991310,2cyclictest6922-21tail20:10:253
8210991310,2cyclictest20863-21cron22:20:003
8210991310,2cyclictest19837-21grep20:40:133
8210991310,2cyclictest1195-21cron21:10:013
8210991310,2cyclictest11448-21tr00:10:133
8210991310,2cyclictest10266-21grep19:10:273
821099130,12cyclictest27452-21head20:55:203
821099130,12cyclictest0-21swapper/321:45:553
820799139,3cyclictest6000-21grep22:30:152
820799139,3cyclictest31696-21grep22:25:262
820799139,3cyclictest16968-21chrt22:55:282
820799139,2cyclictest9641-21irqstats23:30:192
820799139,2cyclictest10928-21ssh23:50:212
8207991310,2cyclictest27913-21sed00:00:222
8207991310,2cyclictest25106-21sort23:20:212
8207991310,2cyclictest25022-21tr20:50:162
820799130,4cyclictest0-21swapper/221:11:002
820799130,0cyclictest0-21swapper/223:17:132
820699135,4cyclictest225-21systemd-journal22:29:551
820699134,2cyclictest30368-21ssh21:26:481
820699132,7cyclictest21486-21kworker/1:120:55:161
820699132,2cyclictest23649-21tr23:00:161
8206991312,1cyclictest0-21swapper/122:35:031
8206991312,1cyclictest0-21swapper/122:35:021
8206991312,1cyclictest0-21swapper/122:12:251
8206991310,2cyclictest4374-21wc22:10:001
8206991310,2cyclictest15072-21grep20:30:171
8206991310,2cyclictest12876-21grep00:30:141
820699130,13cyclictest0-21swapper/122:24:361
820699130,12cyclictest19782-21grep23:55:271
820699130,12cyclictest0-21swapper/100:23:161
820699130,11cyclictest0-21swapper/123:20:151
820599139,3cyclictest8755-21grep23:30:140
820599136,5cyclictest0-21swapper/021:15:120
820599135,6cyclictest5048-21sh21:11:300
8205991312,1cyclictest0-21swapper/021:50:030
8205991311,1cyclictest26447-21sh21:25:040
8205991310,3cyclictest8202-21cyclictest22:58:350
8205991310,2cyclictest6493-21ls20:10:210
8205991310,2cyclictest31883-21sort23:05:140
8205991310,2cyclictest31824-21cat22:45:190
8205991310,2cyclictest28219-21sh22:05:000
821099129,2cyclictest29553-21head22:05:183
821099129,2cyclictest20308-21grep21:20:253
821099129,2cyclictest18200-21cat19:30:003
821099129,2cyclictest11744-21sed20:20:263
821099128,4cyclictest30553-21apache222:58:283
821099128,3cyclictest39-21ksoftirqd/322:40:003
821099128,3cyclictest39-21ksoftirqd/322:40:003
821099128,2cyclictest31317-21sed19:55:173
821099127,4cyclictest13775-21wc00:30:193
821099127,2cyclictest22881-21sendmail19:40:003
821099121,0cyclictest0-21swapper/300:27:133
8210991210,1cyclictest4136-21tr20:05:223
821099120,1cyclictest0-21swapper/322:43:123
820799129,2cyclictest9361-21grep20:15:272
820799129,2cyclictest8313-21ssh23:30:002
820799129,2cyclictest3387-21grep21:30:192
820799129,2cyclictest32-21ksoftirqd/219:15:162
820799129,2cyclictest30753-21grep21:00:262
820799129,2cyclictest26363-21wc23:40:182
820799129,2cyclictest22870-21cron19:40:002
820799129,2cyclictest20366-21grep00:15:182
820799129,2cyclictest15043-21tr20:30:172
820799129,2cyclictest14268-21tr22:15:182
820799129,2cyclictest10170-21grep21:15:142
820799128,3cyclictest29666-21ls19:50:252
820799128,2cyclictest32464-21head23:05:172
820799125,6cyclictest0-21swapper/222:14:292
820799124,6cyclictest27584-21/usr/sbin/munin21:25:192
820799121,10cyclictest6594-21ssh22:50:112
820799120,2cyclictest0-21swapper/200:30:122
820799120,12cyclictest0-21swapper/200:06:052
820799120,12cyclictest0-21swapper/200:06:042
820699129,2cyclictest31442-21cat21:05:001
820699129,2cyclictest25758-21grep19:45:141
820699129,2cyclictest21030-21grep00:15:241
820699129,2cyclictest20865-21basename19:35:141
820699129,2cyclictest13412-21sh21:55:221
820699129,2cyclictest12255-21sed19:15:221
820699128,3cyclictest6357-21ssh21:12:311
820699128,3cyclictest17922-21turbostat.cron21:20:001
820699126,5cyclictest38150irq/131-snd_hda00:25:191
820699126,2cyclictest31599-21sed22:45:171
8206991211,1cyclictest2952-21kworker/1:223:50:051
8206991210,1cyclictest0-21swapper/100:35:261
820699120,8cyclictest2692-21ssh21:30:161
820599129,2cyclictest8414-21head23:10:180
820599129,2cyclictest31874-21wc19:55:210
820599129,2cyclictest29069-21tr22:05:140
820599129,2cyclictest27364-21head00:00:180
820599129,2cyclictest16287-21grep19:25:150
820599129,2cyclictest10311-21tr20:20:140
820599128,3cyclictest5507-21head22:10:180
820599128,3cyclictest24219-21wc19:40:190
820599128,3cyclictest14440-21grep22:15:200
820599128,3cyclictest13992-21head00:30:200
8205991210,1cyclictest29862-21cut21:00:160
820599120,3cyclictest0-21swapper/021:35:140
821099118,2cyclictest4249-21ssh23:46:113
821099118,2cyclictest3062-21expr00:05:163
821099118,2cyclictest3062-21expr00:05:153
821099118,2cyclictest301-21ssh22:45:273
821099118,2cyclictest30072-21taskset19:53:083
821099118,2cyclictest28642-21head21:25:283
821099118,2cyclictest27370-21ssh23:40:313
821099118,2cyclictest21763-21ssh21:40:503
821099118,2cyclictest20694-21sh23:36:483
821099118,2cyclictest17452-21grep20:35:163
821099118,2cyclictest14531-21cron20:30:003
821099118,2cyclictest11640-21ssh23:31:153
821099118,2cyclictest10279-21cat21:15:153
821099118,2cyclictest10122-21ssh22:51:383
821099117,3cyclictest935-21cat20:00:133
821099116,5cyclictest1436-21sendmail-mta22:25:343
820799118,2cyclictest31513-21cat21:05:142
820799118,2cyclictest30747-21ssh22:45:122
820799118,2cyclictest28999-21sort22:05:142
820799118,2cyclictest28367-21sed20:55:292
820799118,2cyclictest21876-21ssh21:40:562
820799118,2cyclictest20746-21ssh00:35:012
820799118,2cyclictest20042-21sed19:30:242
820799118,2cyclictest17011-21ssh23:35:132
820799118,2cyclictest16170-21grep19:25:152
820799118,2cyclictest12171-21apt20:25:012
820799118,2cyclictest11784-21tail21:35:182
820799117,3cyclictest29251-21ssh00:20:222
820799113,4cyclictest225-21systemd-journal20:06:032
820799111,3cyclictest0-21swapper/222:35:402
820799111,3cyclictest0-21swapper/222:35:392
820799110,10cyclictest8562-21rm23:10:182
820699118,2cyclictest8730-21grep19:10:141
820699118,2cyclictest64392sleep122:50:051
820699118,2cyclictest5823-21ssh21:50:401
820699118,2cyclictest4508-21grep20:05:251
820699118,2cyclictest31721-21tr21:05:171
820699118,2cyclictest23645-21head20:45:291
820699118,2cyclictest23645-21head20:45:281
820699118,2cyclictest21212-21ssh21:21:041
820699118,2cyclictest2000-21grep20:00:251
820699118,2cyclictest19202-21taskset20:39:031
820699118,2cyclictest1895-21grep23:45:171
820699118,2cyclictest16196-21ssh23:34:501
820699118,2cyclictest13077-21cut20:25:181
820699118,2cyclictest12180-21ssh21:35:211
820699118,2cyclictest10335-21tr20:20:151
820699117,3cyclictest29151-21wc19:50:191
820699111,9cyclictest0-21swapper/121:40:121
820699110,10cyclictest218242sleep123:18:301
820599118,2cyclictest8429-21grep20:15:170
820599118,2cyclictest5228-21cat22:30:010
820599118,2cyclictest4154-21gzip21:30:250
820599118,2cyclictest4129-21grep20:05:210
820599118,2cyclictest28420-21head21:45:170
820599118,2cyclictest26537-21cat19:45:180
820599118,2cyclictest26457-21ssh00:38:130
820599118,2cyclictest2592-21ssh00:05:090
820599118,2cyclictest2592-21ssh00:05:080
820599118,2cyclictest24378-21ssh22:40:270
820599118,2cyclictest20366-21sed21:20:250
820599118,2cyclictest16450-21ssh22:35:460
820599118,2cyclictest16450-21ssh22:35:450
820599118,2cyclictest14872-21cut20:30:150
820599118,2cyclictest13863-21cat20:25:270
820599118,2cyclictest1302-21sed20:00:160
820599118,2cyclictest12066-21cat19:15:200
820599118,2cyclictest11826-21grep00:10:150
820599117,3cyclictest22229-21grep22:20:180
820599113,4cyclictest225-21systemd-journal19:14:550
821099108,1cyclictest29104-21grep21:00:133
821099108,1cyclictest25627-21gzip20:50:213
821099108,1cyclictest24634-21threads19:40:233
821099108,1cyclictest23126-21open_inodes20:45:233
821099108,1cyclictest23126-21open_inodes20:45:223
821099108,1cyclictest13708-21expr19:20:183
821099107,2cyclictest7847-21wc20:15:143
821099107,2cyclictest26980-21wc19:45:253
821099107,2cyclictest19590-21grep19:30:193
820799108,1cyclictest24217-21sed20:50:012
820799108,1cyclictest24217-21sed20:50:012
820799108,1cyclictest21426-21taskset20:41:522
820799108,1cyclictest192022sleep220:39:032
820799107,2cyclictest5944-21cut20:10:162
820799107,2cyclictest31269-21head19:55:172
820799107,2cyclictest26994-21cat19:45:252
820799100,9cyclictest13588-21grep19:20:162
820699108,1cyclictest8299-21seq20:15:161
820699108,1cyclictest6749-21awk20:10:241
820699108,1cyclictest17284-21chrt19:25:221
820699107,2cyclictest24450-21cut20:50:131
820699107,2cyclictest19642-21wget20:40:131
820699107,2cyclictest19014-21cut19:30:141
820699107,2cyclictest15213-21chrt19:20:521
820599108,1cyclictest24456-21wget20:50:130
820599108,1cyclictest23692-21chrt20:45:310
820599108,1cyclictest23692-21chrt20:45:300
820599108,1cyclictest21713-21cut19:35:190
820599108,1cyclictest20394-21taskset20:40:180
820599107,2cyclictest29341-21grep19:50:220
820599107,2cyclictest26984-21tr20:55:180
820599107,2cyclictest23118-21ssh23:00:050
82069997,1cyclictest32027-21awk19:55:241
82069997,1cyclictest23204-21awk19:40:121
82059997,1cyclictest32674-21cat21:05:220
82059997,1cyclictest17990-21cat20:35:190
82059985,2cyclictest19128-21tail19:30:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional