You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 12:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 31, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
637129380,9sleep00-21swapper/019:08:580
610928766,17sleep20-21swapper/219:05:352
637228467,13sleep10-21swapper/119:08:591
620328068,8sleep30-21swapper/319:06:493
210612770,0sleep10-21swapper/122:39:521
134042690,0sleep30-21swapper/322:35:123
6544992926,2cyclictest39-21ksoftirqd/319:40:013
654299280,27cyclictest0-21swapper/221:18:122
653899241,22cyclictest0-21swapper/119:30:131
6535992412,5cyclictest1142-21runrttasks22:32:370
654299220,22cyclictest0-21swapper/200:08:242
654299220,21cyclictest0-21swapper/222:26:372
654299220,21cyclictest0-21swapper/219:25:122
654299220,1cyclictest0-21swapper/222:46:572
654299220,1cyclictest0-21swapper/221:56:392
654499210,19cyclictest14009-21kworker/3:019:16:363
654299210,2cyclictest0-21swapper/222:32:042
653899210,2cyclictest0-21swapper/120:55:141
6544992018,2cyclictest0-21swapper/319:12:023
654299200,18cyclictest0-21swapper/223:40:432
654299190,18cyclictest0-21swapper/223:35:392
654299190,18cyclictest0-21swapper/200:34:202
654299190,14cyclictest0-21swapper/222:54:032
653899190,1cyclictest0-21swapper/100:28:181
653899190,19cyclictest0-21swapper/100:22:571
653899190,17cyclictest0-21swapper/122:50:361
6535991916,2cyclictest13-21ksoftirqd/019:40:010
653599190,16cyclictest0-21swapper/021:58:530
183962190,1sleep129220-21kworker/1:200:14:121
654299180,14cyclictest0-21swapper/223:51:202
653899182,1cyclictest31479-21diskmemload21:54:561
653899182,1cyclictest31479-21diskmemload21:54:551
653899180,18cyclictest0-21swapper/123:06:481
654499170,12cyclictest0-21swapper/322:10:033
6542991716,1cyclictest0-21swapper/223:04:032
654299170,15cyclictest0-21swapper/223:57:452
6538991716,1cyclictest0-21swapper/121:16:551
654499160,2cyclictest0-21swapper/300:30:553
6542991615,0cyclictest0-21swapper/220:15:162
6542991614,2cyclictest0-21swapper/222:07:532
654299160,16cyclictest0-21swapper/221:14:162
6538991615,1cyclictest0-21swapper/123:00:131
653899160,15cyclictest0-21swapper/100:33:371
653599165,6cyclictest29698-21latency_hist22:44:590
653599160,13cyclictest0-21swapper/022:10:040
6542991512,3cyclictest0-21swapper/222:23:232
654299150,13cyclictest0-21swapper/200:00:252
653899159,5cyclictest13802-21ls22:15:261
6538991514,1cyclictest6839-21kworker/1:122:14:531
6538991514,1cyclictest28985-21kworker/1:223:22:431
6538991514,1cyclictest28985-21kworker/1:223:22:421
6538991513,2cyclictest25-21ksoftirqd/123:16:591
6538991512,2cyclictest19161-21df_abs23:55:181
653899150,1cyclictest0-21swapper/123:12:241
653899150,15cyclictest0-21swapper/100:03:301
653899150,12cyclictest13038-21df20:30:161
653599157,7cyclictest24988-21munin-run23:40:010
6535991510,2cyclictest31023-21sh23:05:000
6544991412,2cyclictest0-21swapper/323:01:363
6544991411,2cyclictest15880-21perf21:20:003
6544991410,3cyclictest8123-21df23:10:173
6544991410,3cyclictest30200-21sed00:20:263
6544991410,3cyclictest16196-21grep22:55:233
6544991410,2cyclictest4427-21wc00:05:243
654499140,1cyclictest0-21swapper/321:37:273
654499140,14cyclictest0-21swapper/323:24:183
654499140,14cyclictest0-21swapper/323:24:173
654499140,13cyclictest7839-21basename20:20:133
654299142,1cyclictest0-21swapper/223:21:292
654299142,1cyclictest0-21swapper/223:21:282
6542991413,1cyclictest0-21swapper/220:05:262
654299140,11cyclictest0-21swapper/222:37:052
6538991413,1cyclictest21359-21kworker/1:022:05:321
6538991413,1cyclictest21359-21kworker/1:022:01:321
6538991410,3cyclictest27472-21grep19:50:261
6538991410,3cyclictest25-21ksoftirqd/122:49:481
6538991410,3cyclictest0-21swapper/121:57:341
653899140,14cyclictest0-21swapper/122:55:301
6535991410,3cyclictest21184-21grep22:00:270
6535991410,2cyclictest4335-21grep00:25:130
654499139,3cyclictest29916-21grep21:05:183
654499139,3cyclictest19771-21grep20:45:133
654499139,3cyclictest14609-21turbostat20:35:003
654499139,3cyclictest1226-21ssh21:10:253
6544991310,2cyclictest591-21grep20:05:133
6544991310,2cyclictest27703-21cron00:20:003
6544991310,2cyclictest12539-21grep21:55:283
6544991310,2cyclictest10089-21cron21:55:013
6544991310,2cyclictest10089-21cron21:55:003
654299139,3cyclictest25313-21sh21:25:142
654299139,3cyclictest14605-21turbostat.cron20:35:002
654299139,3cyclictest0-21swapper/200:10:122
654299139,2cyclictest4829-21taskset22:10:232
654299132,10cyclictest56550irq/124-enp0s3119:50:172
654299131,4cyclictest56550irq/124-enp0s3120:01:312
654299131,1cyclictest14195-21ssh23:33:082
6542991311,1cyclictest0-21swapper/221:45:132
6542991310,2cyclictest17976-21head20:40:182
6542991310,2cyclictest12320-21grep19:20:172
654299130,2cyclictest0-21swapper/223:05:162
654299130,12cyclictest0-21swapper/200:15:142
654299130,11cyclictest0-21swapper/200:26:532
653899139,3cyclictest4221-21wc20:10:211
653899139,3cyclictest21768-21grep19:40:161
653899139,3cyclictest19033-21tr21:40:201
653899136,3cyclictest225-21systemd-journal20:35:261
6538991312,1cyclictest27095-21ssh23:40:231
6538991312,1cyclictest0-21swapper/119:45:131
6538991311,1cyclictest0-21swapper/100:07:591
6538991310,2cyclictest21765-21head22:20:211
6538991310,2cyclictest12317-21head19:20:171
653899130,1cyclictest55650irq/127-enp2s0-23:27:421
653899130,1cyclictest5518-21kworker/1:023:36:431
653899130,12cyclictest969-21head21:10:231
653599139,3cyclictest9622-21cron23:50:000
653599139,3cyclictest31620-21grep23:05:140
653599139,2cyclictest9287-21grep19:15:140
653599136,6cyclictest0-21swapper/022:45:130
6535991310,2cyclictest3587-21grep20:10:160
6535991310,2cyclictest2651-21awk20:10:010
6535991310,2cyclictest21980-21cut20:50:000
6535991310,2cyclictest17385-21cat21:40:010
653599130,2cyclictest141rcu_preempt00:13:450
653599130,12cyclictest9507-21head23:30:170
653599130,11cyclictest0-21swapper/022:05:130
654499129,2cyclictest9034-21grep23:30:153
654499129,2cyclictest6677-21grep22:50:143
654499129,2cyclictest6162-21ssh00:25:273
654499129,2cyclictest5284-21cut22:30:153
654499129,2cyclictest30498-21cat20:00:003
654499129,2cyclictest2900-21grep21:30:263
654499129,2cyclictest28997-21cut22:25:133
654499129,2cyclictest17834-21cut23:35:183
654499129,2cyclictest17470-21tail21:20:183
654499129,2cyclictest15105-21cut20:35:153
654499129,2cyclictest10369-21tr20:25:163
654499129,1cyclictest0-21swapper/322:16:593
654499128,3cyclictest25858-21tail21:25:163
654499128,3cyclictest17906-21sed19:30:273
654499126,2cyclictest225-21systemd-journal19:25:003
6544991210,1cyclictest6583-21grep20:15:223
654299129,2cyclictest25645-21ssh21:45:002
654299129,2cyclictest22498-21chrt22:01:252
654299129,2cyclictest21958-21grep22:40:142
654299129,2cyclictest19436-21sed19:35:172
654299129,2cyclictest17311-21ls23:15:222
654299129,2cyclictest12137-21basename22:15:142
654299126,6cyclictest3338-21apache223:46:282
653899129,2cyclictest6437-21sed20:15:201
653899129,2cyclictest578-21cat21:30:001
653899129,2cyclictest3608-21grep23:45:271
653899129,2cyclictest32684-21sh20:05:011
653899129,2cyclictest24322-21cron20:54:591
653899129,2cyclictest22467-21grep00:35:231
653899129,2cyclictest2215-21tr21:30:191
653899129,2cyclictest16959-21grep21:20:161
653899128,3cyclictest5178-21wget22:30:141
653899127,1cyclictest30259-21if_err_enp2s021:05:201
653899120,2cyclictest0-21swapper/122:26:151
653599129,2cyclictest7476-21head19:10:170
653599129,2cyclictest31467-21cron21:10:000
653599129,2cyclictest3010-21grep21:50:170
653599129,2cyclictest3010-21grep21:50:160
653599129,2cyclictest25794-21wc23:20:190
653599129,2cyclictest25794-21wc23:20:190
653599129,2cyclictest25562-21basename19:50:000
653599129,2cyclictest24712-21basename20:55:150
653599129,2cyclictest19453-21sed23:55:200
653599129,2cyclictest17179-21grep23:15:200
653599129,2cyclictest15892-21tr21:19:590
653599129,2cyclictest13883-21sed22:15:260
653599129,2cyclictest13582-21ssh22:35:160
653599129,2cyclictest11176-21ssh23:50:190
653599128,2cyclictest28539-21grep00:20:150
653599120,11cyclictest26223-21cat21:25:190
280872120,0sleep00-21swapper/000:39:240
654499118,2cyclictest5086-21chrt22:49:243
654499118,2cyclictest32506-21basename23:05:183
654499118,2cyclictest30613-21tr20:00:143
654499118,2cyclictest29534-21sed22:05:273
654499118,2cyclictest28837-21chrt21:01:483
654499118,2cyclictest27732-21taskset19:52:453
654499118,2cyclictest26897-21grep21:45:173
654499118,2cyclictest26706-21ls20:59:593
654499118,2cyclictest2410-21cut23:45:183
654499118,2cyclictest23362-21cut22:40:223
654499118,2cyclictest21770-21head19:40:163
654499118,2cyclictest20680-21ssh23:17:343
654499118,2cyclictest20613-21ssh21:41:043
654499118,2cyclictest20337-21wc22:00:193
654499118,2cyclictest20223-21chrt22:20:073
654499118,2cyclictest19358-21tune2fs23:55:193
654499118,2cyclictest15619-21sed19:25:263
654499118,2cyclictest13138-21ssh00:10:293
654499118,2cyclictest1167-21sed23:25:183
654499113,5cyclictest21037-21apt-get00:35:143
654499110,10cyclictest28046-21sh23:40:513
654299118,3cyclictest0-21swapper/200:24:222
654299118,2cyclictest7734-21ssh21:34:062
654299118,2cyclictest4652-21sed20:10:262
654299118,2cyclictest28893-21tail19:55:172
654299118,2cyclictest24119-21ssh00:36:182
654299118,2cyclictest23033-21idleruntime20:50:182
654299118,2cyclictest18231-21taskset19:32:572
654299118,2cyclictest11911-21ssh21:35:512
654299118,2cyclictest11105-21chrt19:18:092
654299113,3cyclictest225-21systemd-journal23:15:012
654299111,5cyclictest0-21swapper/222:55:122
654299110,3cyclictest0-21swapper/221:50:062
654299110,3cyclictest0-21swapper/221:50:062
654299110,0cyclictest0-21swapper/219:45:142
653899118,2cyclictest9621-21grep21:35:151
653899118,2cyclictest9172-21cat23:30:151
653899118,2cyclictest7868-21grep19:10:201
653899118,2cyclictest29408-21ssh21:46:241
653899118,2cyclictest27052-21ssh22:42:581
653899118,2cyclictest16020-21cron19:29:591
653899118,2cyclictest12183-21ssh23:50:291
653899118,2cyclictest11285-21readlink19:19:591
653899117,3cyclictest807-21cut20:05:141
653899117,3cyclictest18000-21head20:40:181
653899115,4cyclictest20100-21aten2_r7power_v00:15:151
653899114,6cyclictest10192-21cut20:25:161
653599118,2cyclictest6903-21ssh21:14:340
653599118,2cyclictest6577-21ssh00:06:430
653599118,2cyclictest6137-21which20:15:180
653599118,2cyclictest32730-21date20:05:010
653599118,2cyclictest27409-21ls19:50:250
653599118,2cyclictest26787-21ssh00:00:090
653599118,2cyclictest26511-21cat23:40:180
653599118,2cyclictest21236-21ssh21:22:290
653599118,2cyclictest20218-21grep00:15:150
653599118,2cyclictest18871-21ssh00:33:450
653599118,2cyclictest17639-21ssh21:40:060
653599118,2cyclictest1711-21cut21:30:160
653599118,2cyclictest12127-21ssh23:12:260
653599118,2cyclictest11120-21ssh22:52:340
654499108,1cyclictest4771-21chrt20:11:163
654499108,1cyclictest27425-21expr00:00:153
654499108,1cyclictest242802sleep320:54:313
654499108,1cyclictest23455-21cut19:45:153
654499107,2cyclictest9928-21ssh23:50:033
654499100,9cyclictest18226-21cat20:40:203
654299108,1cyclictest7410-21grep19:10:162
654299108,1cyclictest31177-21taskset21:06:212
654299108,1cyclictest27646-21grep21:00:162
654299108,1cyclictest21406-21expr19:40:142
654299108,1cyclictest19456-21ssh21:21:062
654299107,2cyclictest7745-21chrt20:20:112
654299107,2cyclictest32447-21ssh23:25:012
654299107,2cyclictest24803-21cat20:55:152
654299107,2cyclictest19845-21cpuspeed_turbos20:45:132
653899108,1cyclictest7729-21bc20:20:101
653899108,1cyclictest30025-21chrt19:56:421
653899108,1cyclictest21701-21taskset20:49:371
653899108,1cyclictest18981-21wc19:35:151
653599108,2cyclictest13729-21awk20:30:220
653599108,1cyclictest8853-21uname20:20:190
653599108,1cyclictest28876-21ssh22:25:110
653599108,1cyclictest27597-21grep21:00:160
653599108,1cyclictest25892-21ssh21:45:010
653599108,1cyclictest21963-21cat19:40:180
653599108,1cyclictest20118-21ssh22:20:020
653599108,1cyclictest167112sleep020:36:440
653599108,1cyclictest14008-21grep19:25:150
653599107,2cyclictest32546-21ssh23:25:070
653599107,2cyclictest28417-21grep19:55:150
653599107,2cyclictest18196-21uname20:40:200
653599107,2cyclictest18066-21chrt19:30:490
653599107,2cyclictest14706-21ssh22:55:100
653599107,2cyclictest12521-21cat19:20:190
653599107,2cyclictest11358-21tr20:25:220
65429997,1cyclictest15069-21cat20:35:142
65429990,7cyclictest1142-21runrttasks20:25:202
65389997,1cyclictest27703-21expr21:00:161
65359991,4cyclictest225-21systemd-journal20:50:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional