You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 13:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 11, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2636929750,40sleep20-21swapper/219:06:362
2646429669,9sleep30-21swapper/319:07:503
2630029279,9sleep00-21swapper/019:05:410
2649728564,6sleep10-21swapper/119:08:151
50172630,2sleep15013-21ssh23:35:171
2672299270,26cyclictest0-21swapper/223:22:572
2672599250,1cyclictest0-21swapper/321:00:123
2672099250,24cyclictest0-21swapper/121:00:121
26716992418,2cyclictest225-21systemd-journal00:07:220
2672299230,23cyclictest0-21swapper/200:01:002
2672099231,4cyclictest141rcu_preempt21:15:411
2672599221,21cyclictest19519-21diskmemload23:26:383
2672099221,1cyclictest141rcu_preempt23:19:481
26716992212,6cyclictest29594-21apache223:53:010
26716992120,1cyclictest0-21swapper/000:02:580
26725992017,3cyclictest39-21ksoftirqd/322:29:333
2672299200,2cyclictest0-21swapper/223:45:252
2671699200,19cyclictest0-21swapper/023:05:320
2671699200,0cyclictest0-21swapper/022:12:210
2672299190,1cyclictest0-21swapper/221:16:242
2672099190,17cyclictest0-21swapper/100:05:071
2672599182,1cyclictest19519-21diskmemload23:33:373
26725991816,1cyclictest0-21swapper/323:43:403
26725991816,1cyclictest0-21swapper/323:22:033
26722991818,0cyclictest0-21swapper/221:13:152
26722991817,1cyclictest0-21swapper/223:15:512
2672299180,18cyclictest0-21swapper/221:33:452
26720991812,5cyclictest10476-21kworker/1:221:55:141
2672599171,2cyclictest27410-21ssh00:27:083
2672299179,7cyclictest0-21swapper/223:55:202
26722991717,0cyclictest0-21swapper/221:24:412
2672299170,17cyclictest0-21swapper/221:51:402
2672099170,10cyclictest0-21swapper/122:53:031
2672599164,6cyclictest1742-21apt-get19:25:123
2672299160,16cyclictest0-21swapper/200:16:212
2672099163,12cyclictest30706-21mailstats23:50:211
26720991613,3cyclictest0-21swapper/122:15:571
2672099161,14cyclictest11816-21ssh21:24:441
2671699164,2cyclictest225-21systemd-journal23:32:380
26716991614,1cyclictest0-21swapper/022:15:180
2671699160,15cyclictest0-21swapper/022:40:120
2672599156,1cyclictest0-21swapper/322:23:333
2672599151,9cyclictest0-21swapper/323:15:153
2672599151,3cyclictest0-21swapper/320:45:133
2672299154,5cyclictest225-21systemd-journal21:05:002
26722991515,0cyclictest0-21swapper/200:12:072
26722991510,5cyclictest26520-21apache200:27:282
26720991514,1cyclictest0-21swapper/100:23:561
2672099150,15cyclictest0-21swapper/100:01:301
26716991512,2cyclictest30512-21cron20:25:000
2672599148,5cyclictest26520-21apache200:37:293
2672599141,2cyclictest0-21swapper/322:00:133
2672599141,12cyclictest27693-21sed22:50:283
2672599140,14cyclictest0-21swapper/323:53:103
2672599140,13cyclictest0-21swapper/322:57:513
2672299140,14cyclictest0-21swapper/221:39:132
2672299140,14cyclictest0-21swapper/221:39:122
2672099149,4cyclictest683-21seq23:32:451
2672099149,2cyclictest30836-21expr00:10:131
2672099145,4cyclictest0-21swapper/120:10:131
2672099142,4cyclictest19519-21diskmemload22:55:261
26720991412,2cyclictest25-21ksoftirqd/123:05:221
2672099141,13cyclictest7880-21rm22:39:281
2672099140,12cyclictest0-21swapper/121:14:401
2671699149,3cyclictest26390-21wc22:30:240
2671699148,6cyclictest26502-21apache200:25:220
2671699148,2cyclictest225-21systemd-journal00:37:170
26716991411,2cyclictest28751-21wc23:10:240
26716991410,3cyclictest8991-21uname20:45:210
26716991410,3cyclictest32337-21grep00:10:250
2672599138,4cyclictest13027-21expr21:25:143
2672599134,9cyclictest19519-21diskmemload21:43:193
2672599133,4cyclictest19519-21diskmemload21:36:353
2672599133,4cyclictest19519-21diskmemload21:36:343
2672599131,9cyclictest19519-21diskmemload22:15:103
26725991312,1cyclictest0-21swapper/323:05:303
2672599131,12cyclictest39-21ksoftirqd/321:53:153
26725991310,2cyclictest22709-21grep00:25:003
26725991310,2cyclictest18040-21grep19:55:263
26725991310,2cyclictest16178-21cut22:05:183
2672299139,3cyclictest27437-21cat20:15:252
26722991311,2cyclictest32-21ksoftirqd/223:38:402
26722991310,2cyclictest7678-21grep00:35:142
26722991310,2cyclictest32000-21ls20:25:212
26722991310,2cyclictest29919-21cat20:20:252
26722991310,2cyclictest28400-21tr23:10:192
26722991310,2cyclictest26115-21ssh00:07:092
26722991310,2cyclictest22724-21cat00:25:002
26722991310,2cyclictest21331-21grep20:05:142
26722991310,2cyclictest18657-21wc21:05:232
26722991310,2cyclictest18204-21grep22:45:152
26722991310,2cyclictest13499-21uname23:40:182
2672299130,12cyclictest7964-21cat19:35:242
2672299130,12cyclictest0-21swapper/222:26:162
2672099139,3cyclictest38150irq/131-snd_hda21:34:201
2672099138,4cyclictest12078-21basename19:45:181
2672099137,5cyclictest26502-21apache200:37:291
2672099134,4cyclictest5589-21apt-get21:40:131
2672099132,7cyclictest19519-21diskmemload23:25:411
2672099132,1cyclictest0-21swapper/119:20:121
2672099131,4cyclictest0-21swapper/122:03:461
26720991311,2cyclictest0-21swapper/100:29:271
26720991310,2cyclictest8424-21ssh00:15:311
2672099130,12cyclictest28605-21sensors23:10:231
2671699139,3cyclictest22887-21head20:05:250
2671699139,3cyclictest19464-21tune2fs20:00:170
2671699139,3cyclictest11321-21grep19:45:140
2671699138,3cyclictest18359-21wc21:05:200
2671699131,7cyclictest9631-21perf23:00:010
26716991310,2cyclictest8599-21tail22:20:160
26716991310,2cyclictest7467-21grep21:40:260
26716991310,2cyclictest28235-21wc20:19:590
26716991310,2cyclictest1514-21cut20:30:150
26716991310,2cyclictest1468-21cron19:25:010
2671699130,1cyclictest0-21swapper/022:37:070
2672599129,2cyclictest8696-21ssh23:37:283
2672599129,2cyclictest8280-21chrt19:36:053
2672599129,2cyclictest29060-21ls19:15:003
2672599129,2cyclictest27673-21grep20:15:273
2672599129,2cyclictest27270-21tr23:10:123
2672599129,2cyclictest25318-21sed22:30:153
2672599129,2cyclictest23425-21ssh22:10:103
2672599129,2cyclictest23181-21chrt20:08:363
2672599129,2cyclictest22098-21grep00:05:143
2672599129,2cyclictest17529-21ssh23:04:323
2672599129,2cyclictest17214-21ssh22:45:043
2672599129,2cyclictest15856-21ssh22:44:123
2672599129,2cyclictest14099-21cron00:20:013
2672599129,2cyclictest13272-21cut20:55:173
2672599129,2cyclictest11887-21grep20:50:253
2672599127,4cyclictest20775-21apt-get23:45:133
2672599126,3cyclictest19519-21diskmemload21:55:143
2672599123,4cyclictest28356-21apt-get20:20:123
26725991211,1cyclictest0-21swapper/321:30:123
2672599120,5cyclictest0-21swapper/300:33:203
2672599120,3cyclictest0-21swapper/323:55:143
2672599120,1cyclictest141rcu_preempt00:01:413
2672299129,2cyclictest610-21wc19:20:212
2672299129,2cyclictest28550-21grep19:10:232
2672299129,2cyclictest17656-21grep19:55:222
2672299129,2cyclictest11134-21cat19:45:012
2672299128,3cyclictest1315-21grep22:15:262
2672299126,1cyclictest225-21systemd-journal23:09:372
2672299124,4cyclictest225-21systemd-journal20:39:592
2672299120,3cyclictest0-21swapper/222:38:492
2672299120,2cyclictest0-21swapper/223:51:032
2672299120,1cyclictest0-21swapper/223:25:222
2672299120,1cyclictest0-21swapper/222:24:482
2672299120,11cyclictest23675-21wc22:10:142
2672099129,3cyclictest15481-21sh23:02:551
2672099129,2cyclictest8853-21ssh22:40:021
2672099129,2cyclictest5225-21cat20:39:591
2672099129,2cyclictest31701-21date19:20:001
2672099129,2cyclictest20511-21rm23:45:021
2672099125,6cyclictest16385-21apt-get19:55:031
2672099124,3cyclictest0-21swapper/119:10:121
26720991211,1cyclictest19519-21diskmemload23:22:281
2672099120,3cyclictest0-21swapper/122:30:041
2672099120,1cyclictest0-21swapper/120:05:161
2672099120,11cyclictest8314-21rm23:56:231
2672099120,11cyclictest7918-21tr20:45:141
2672099120,11cyclictest18218-21grep22:45:161
2671699129,2cyclictest6234-21head20:40:160
2671699129,2cyclictest5073-21sort19:30:170
2671699129,2cyclictest4351-21tr23:35:150
2671699129,2cyclictest32283-21tail00:30:160
2671699129,2cyclictest28047-21grep19:10:180
2671699129,2cyclictest21937-21sed21:10:260
2671699129,2cyclictest19519-21diskmemload21:35:220
2671699129,2cyclictest19519-21diskmemload21:35:210
2671699129,2cyclictest16670-21grep21:00:250
2671699129,2cyclictest16244-21grep22:05:170
2671699129,2cyclictest15532-21cut00:20:160
2671699129,2cyclictest14113-21sendmail00:20:000
2671699129,2cyclictest13246-21grep23:40:160
2671699129,2cyclictest11529-21cut23:20:140
2671699128,3cyclictest8224-21head22:00:220
2671699124,4cyclictest225-21systemd-journal21:31:410
26716991210,1cyclictest19519-21diskmemload23:55:140
2671699120,5cyclictest0-21swapper/021:16:420
2671699120,1cyclictest0-21swapper/022:53:570
2671699120,12cyclictest0-21swapper/023:02:350
200542120,0chrt0-21swapper/022:26:560
2672599119,1cyclictest6496-21wc20:40:203
2672599119,1cyclictest50432sleep320:37:523
2672599119,1cyclictest31135-21ssh00:10:153
2672599119,1cyclictest2849-21ssh22:35:323
2672599119,1cyclictest14295-21cat21:45:143
2672599119,1cyclictest13804-21taskset19:50:083
2672599119,1cyclictest11922sleep319:21:353
2672599118,2cyclictest4156-21cut19:30:133
2672599118,2cyclictest31460-21cut20:25:153
2672599118,2cyclictest30536-21ssh21:15:313
2672599111,5cyclictest0-21swapper/321:20:123
26725991111,0cyclictest0-21swapper/321:11:193
2672299119,1cyclictest6243-21grep20:40:162
2672299118,2cyclictest6249-21forks21:40:162
2672299118,2cyclictest5794-21sed19:30:252
2672299118,2cyclictest30640-21ssh21:55:112
2672299118,2cyclictest28881-21grep23:30:162
2672299118,2cyclictest27356-21basename22:50:232
2672299118,2cyclictest16934-21cat22:05:252
2672299118,2cyclictest14886-21missed_timers19:50:182
2672299118,2cyclictest13522-21cut21:25:162
2672299118,2cyclictest12750-21ssh22:41:442
2672299118,2cyclictest11704-21grep23:00:212
2672299118,2cyclictest11286-21grep19:45:132
2672299118,2cyclictest10028-21rm22:01:222
2672299117,3cyclictest9361-21cat20:45:252
2672299116,5cyclictest29584-21apache222:59:572
2672299115,1cyclictest225-21systemd-journal22:34:372
26722991110,1cyclictest14119-21tr21:45:142
2672099119,2cyclictest29513-21ntpq20:20:201
2672099119,2cyclictest14643-21chrt20:58:581
2672099119,1cyclictest30787-21apt20:25:111
2672099119,1cyclictest23880-21ssh21:50:221
2672099119,1cyclictest19066-21grep20:00:141
2672099119,1cyclictest16540-21ssh21:26:551
2672099119,1cyclictest16478-21ssh21:45:511
2672099119,1cyclictest12906-21expr23:40:141
2672099118,2cyclictest7741-21cat22:20:101
2672099118,2cyclictest5574-21wget20:40:131
2672099118,2cyclictest32552-21ssh21:36:181
2672099118,2cyclictest32552-21ssh21:36:181
2672099118,2cyclictest26699-21grep20:15:161
2672099118,2cyclictest15870-21grep22:05:151
2672099118,2cyclictest14275-21expr19:50:141
2672099118,2cyclictest10270-21sensors19:40:231
2672099113,3cyclictest19519-21diskmemload22:27:201
2672099110,1cyclictest0-21swapper/119:36:331
2672099110,1cyclictest0-21swapper/100:30:231
2672099110,10cyclictest23310-21sh22:10:051
2671699118,2cyclictest9687-21ssh23:19:040
2671699118,2cyclictest9290-21sh21:22:440
2671699118,2cyclictest6568-21cut19:35:130
2671699118,2cyclictest32422-21tail20:25:250
2671699118,2cyclictest22627-21ssh22:47:560
2671699118,2cyclictest22591-21cut21:50:140
2671699118,2cyclictest21494-21uniq23:45:170
2671699118,2cyclictest15161-21grep19:50:210
2671699118,2cyclictest14355-21cat21:45:150
2671699118,2cyclictest1280-21ssh21:56:120
2671699117,3cyclictest2726-21wc19:25:170
2671699116,2cyclictest225-21systemd-journal20:00:010
2672599108,1cyclictest9255-21expr19:40:153
2672599108,1cyclictest29431-21wget19:15:153
2672599108,1cyclictest19499-21cut20:00:173
2672599108,1cyclictest18710-21grep21:05:223
2672599108,1cyclictest1063-21cat20:30:133
2672599107,2cyclictest23824-21tr20:10:143
2672299108,1cyclictest3480-21chrt19:25:262
2672299108,1cyclictest24392-21basename20:10:182
2672299108,1cyclictest11948-21sed20:50:252
2672299107,2cyclictest32281-21grep00:30:152
2672299107,2cyclictest2845-21latency_hist20:35:002
2672299107,2cyclictest14367-21vmstat20:55:272
2672099109,1cyclictest18926-21sed21:05:271
2672099109,1cyclictest1814-21aten2_r7power_p19:25:121
2672099109,1cyclictest10968-21expr20:50:161
2672099108,1cyclictest738-21basename20:30:121
2672099108,1cyclictest5885-21taskset19:30:501
2671699108,1cyclictest253842sleep020:11:170
2671699108,1cyclictest20533-21gltestperf23:25:170
2671699108,1cyclictest12769-21grep20:55:150
2671699107,2cyclictest5026-21taskset20:37:390
2671699107,2cyclictest12645-21ssh21:25:110
2671699107,2cyclictest10536-21tr20:50:130
2671699107,2cyclictest10203-21wc19:40:220
267259998,1cyclictest11712-21ls19:45:143
267229997,1cyclictest29473-21egrep19:15:142
267229994,2cyclictest452-21systemd-logind20:00:232
267169997,1cyclictest31184-21taskset19:16:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional