You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 04:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Jul 26, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24082210164,33sleep00-21swapper/019:05:500
2429329368,8sleep30-21swapper/319:08:303
2416428875,9sleep20-21swapper/219:06:522
2349727867,6sleep10-21swapper/119:05:221
99052550,1sleep1241rcuc/122:41:001
2449499261,1cyclictest3343-21/usr/sbin/munin21:40:231
2449099261,7cyclictest14306-21ntp_states22:05:230
2449999240,23cyclictest0-21swapper/223:11:422
2449999230,23cyclictest0-21swapper/222:22:522
2449999230,23cyclictest0-21swapper/222:22:522
2449999230,23cyclictest0-21swapper/221:26:552
2449999230,21cyclictest0-21swapper/221:54:432
2449999230,21cyclictest0-21swapper/200:10:082
2449999230,1cyclictest0-21swapper/222:29:412
24494992322,1cyclictest0-21swapper/122:06:371
24490992313,5cyclictest32353-21/usr/sbin/munin00:30:230
2449999220,22cyclictest0-21swapper/223:09:062
2449499220,21cyclictest0-21swapper/100:38:481
2450199211,18cyclictest25789-21sshd20:19:273
2449999211,1cyclictest141rcu_preempt22:15:072
24490992120,1cyclictest0-21swapper/022:29:410
2449999200,19cyclictest0-21swapper/223:47:132
24494992018,1cyclictest27699-21cut21:35:161
2449499201,19cyclictest0-21swapper/123:55:001
2449499200,0cyclictest0-21swapper/123:42:081
24490992019,1cyclictest0-21swapper/023:11:430
2449099200,20cyclictest0-21swapper/022:55:420
2449999190,19cyclictest0-21swapper/223:15:352
2449999190,18cyclictest0-21swapper/223:37:582
2449499190,18cyclictest0-21swapper/100:27:281
24490991918,1cyclictest0-21swapper/022:38:350
24490991911,6cyclictest13-21ksoftirqd/000:24:410
24501991815,2cyclictest17830-21kworker/3:021:51:023
2449999180,18cyclictest0-21swapper/222:30:252
2449999180,18cyclictest0-21swapper/222:00:282
2449499180,18cyclictest0-21swapper/100:07:101
2449499180,18cyclictest0-21swapper/100:07:101
2449099180,2cyclictest0-21swapper/022:32:530
2449999170,2cyclictest141rcu_preempt23:35:002
2449999170,16cyclictest0-21swapper/222:49:352
2449499172,15cyclictest0-21swapper/100:03:401
24494991715,2cyclictest8151-21latency_hist23:00:011
2449499170,16cyclictest3887-21cut20:40:171
24490991716,1cyclictest0-21swapper/023:01:040
24490991711,5cyclictest13-21ksoftirqd/021:26:170
2449099170,1cyclictest0-21swapper/021:44:060
2450199161,1cyclictest0-21swapper/300:34:453
2449999160,3cyclictest0-21swapper/200:03:552
2449999160,15cyclictest0-21swapper/200:23:402
2449499161,14cyclictest8178-21ssh22:01:421
2449499160,2cyclictest0-21swapper/122:19:421
2449499160,11cyclictest10888-21head23:20:181
2450199151,6cyclictest26962-21sed21:35:003
24501991511,3cyclictest18244-21kworker/3:221:22:423
2449999150,1cyclictest0-21swapper/223:41:182
2449999150,13cyclictest0-21swapper/200:06:302
2449999150,13cyclictest0-21swapper/200:06:302
24494991511,3cyclictest23207-21cstates22:30:181
2449499150,1cyclictest0-21swapper/123:00:151
2449499150,14cyclictest25171-21ls23:10:001
2449499150,14cyclictest0-21swapper/121:10:511
2449099158,6cyclictest17293-21munin-run21:09:590
2449099153,11cyclictest30878-21apt-get20:30:140
2449099153,11cyclictest30878-21apt-get20:30:140
2449099151,11cyclictest0-21swapper/023:20:140
2449099150,13cyclictest0-21swapper/000:15:220
2450199149,3cyclictest25275-21ssh22:11:473
2450199145,4cyclictest19789-21/usr/sbin/munin23:45:263
24501991411,2cyclictest19260-21tr20:05:163
24501991411,2cyclictest16552-21ssh22:25:353
24501991410,2cyclictest8933-21cat22:40:263
2450199140,14cyclictest0-21swapper/323:32:413
24499991412,2cyclictest32-21ksoftirqd/222:35:152
24499991410,1cyclictest17294-21diskmemload21:56:032
2449999140,4cyclictest0-21swapper/223:23:332
2449999140,14cyclictest0-21swapper/223:01:512
2449999140,14cyclictest0-21swapper/222:41:122
24494991412,1cyclictest0-21swapper/122:13:131
24494991410,3cyclictest13055-21grep21:00:161
2449499140,12cyclictest31587-21tail20:30:181
2449499140,12cyclictest31587-21tail20:30:171
24490991412,2cyclictest13-21ksoftirqd/021:12:100
2449099141,1cyclictest141rcu_preempt00:05:300
2449099141,1cyclictest141rcu_preempt00:05:290
2450199139,3cyclictest2528-21apache223:57:273
2450199139,2cyclictest30012-21grep22:15:153
2450199138,3cyclictest28744-21sed19:15:273
24501991312,1cyclictest0-21swapper/322:03:103
24501991311,1cyclictest5076-21cut19:35:173
24501991310,2cyclictest627-21expr22:55:153
24501991310,2cyclictest3330-21grep19:30:213
24501991310,2cyclictest30281-21grep00:10:153
24501991310,2cyclictest2434-21tail23:15:193
24501991310,2cyclictest22126-21grep20:10:183
24501991310,2cyclictest21330-21grep00:05:183
24501991310,2cyclictest21330-21grep00:05:173
24501991310,2cyclictest14720-21tail19:55:193
2449999131,11cyclictest7180-21tr19:40:292
24499991310,2cyclictest29206-21wc21:35:282
2449999130,7cyclictest0-21swapper/221:15:162
2449999130,1cyclictest0-21swapper/200:39:562
2449999130,13cyclictest0-21swapper/222:54:172
2449499139,3cyclictest6115-21tr00:15:181
2449499139,3cyclictest3735-21sh23:55:001
2449499139,3cyclictest26876-21grep23:30:141
2449499138,3cyclictest26274-21cron23:30:001
24494991312,1cyclictest0-21swapper/121:26:411
24494991310,2cyclictest28095-21wc19:15:201
24494991310,2cyclictest26835-21grep20:20:181
24494991310,2cyclictest2335-21sort23:15:191
24494991310,2cyclictest15886-21wc00:20:261
24494991310,2cyclictest1175-21wc19:25:261
2449499130,12cyclictest30970-21grep00:10:201
2449099139,2cyclictest13-21ksoftirqd/022:54:550
2449099136,6cyclictest25858-21munin-run20:20:000
2449099136,4cyclictest21582-21sh21:30:540
2449099132,9cyclictest0-21swapper/022:47:290
24490991312,1cyclictest0-21swapper/022:40:140
24490991310,2cyclictest8798-21sed20:50:180
24490991310,2cyclictest7262-21grep20:45:270
24490991310,2cyclictest3955-21grep20:40:160
2449099130,13cyclictest0-21swapper/023:47:130
2450199129,2cyclictest8990-21grep00:35:283
2450199129,2cyclictest32332-21tr19:25:153
2450199129,2cyclictest31809-21tr20:30:203
2450199129,2cyclictest31809-21tr20:30:203
2450199129,2cyclictest30473-21head19:20:173
2450199129,2cyclictest17929-21cut21:10:143
2450199129,2cyclictest14212-21cat21:00:273
2450199129,2cyclictest12838-21grep23:40:223
2450199128,3cyclictest18833-21expr23:25:133
2450199128,3cyclictest1373-21sed20:35:163
2450199128,2cyclictest9528-21tr19:45:163
2450199127,4cyclictest24762-21sed22:30:283
2450199126,4cyclictest25478-21/usr/sbin/munin23:10:163
2450199121,9cyclictest17294-21diskmemload21:45:143
2450199120,11cyclictest28813-21tr21:55:133
2450199120,10cyclictest0-21swapper/323:50:143
2449999129,2cyclictest31096-21tr20:30:162
2449999129,2cyclictest31096-21tr20:30:162
2449999129,2cyclictest25675-21munin-run21:15:002
2449999129,2cyclictest10678-21grep20:55:162
2449999127,2cyclictest30609-21apt20:30:002
2449999127,2cyclictest0-21swapper/222:58:152
24499991210,2cyclictest32-21ksoftirqd/223:57:022
24499991210,1cyclictest32-21ksoftirqd/221:45:142
24499991210,1cyclictest23867-21grep00:25:222
2449999120,2cyclictest0-21swapper/200:31:172
2449999120,11cyclictest4641-21grep21:40:212
2449999120,11cyclictest20793-21apt-get22:10:002
2449499129,2cyclictest4181-21grep21:20:281
2449499129,2cyclictest28802-21tr21:55:141
2449499129,2cyclictest13313-21grep19:50:251
2449499129,2cyclictest12650-21cut21:45:171
2449499129,2cyclictest0-21swapper/121:53:151
2449499128,1cyclictest24978-21ssh21:33:321
2449499127,4cyclictest9268-21ssh23:38:541
2449499127,4cyclictest17220-21taskset21:09:221
2449499127,1cyclictest17294-21diskmemload23:45:171
2449499120,2cyclictest0-21swapper/122:52:491
2449099129,2cyclictest7591-21grep19:40:320
2449099129,2cyclictest30614-21dump-pmu-power20:30:000
2449099129,2cyclictest29306-21cat23:50:200
2449099129,2cyclictest19255-21cut20:05:150
2449099129,2cyclictest14586-21grep00:00:280
2449099129,2cyclictest14445-21grep21:00:280
2449099129,2cyclictest12596-21grep21:45:180
2449099129,2cyclictest12193-21cat23:59:590
2449099129,2cyclictest1085-21tr20:35:160
2449099127,4cyclictest17294-21diskmemload00:13:400
2449099125,6cyclictest17294-21diskmemload22:00:190
2449099123,4cyclictest225-21systemd-journal19:20:000
2449099122,5cyclictest0-21swapper/021:15:160
24490991210,1cyclictest2849-21run-parts21:40:000
24490991210,1cyclictest21468-21ntpq21:50:220
2449099120,11cyclictest27608-21grep23:30:180
2450199118,2cyclictest9863-21ssh23:20:073
2450199118,2cyclictest8386-21ssh22:20:453
2450199118,2cyclictest8386-21ssh22:20:453
2450199118,2cyclictest6958-21wget19:40:293
2450199118,2cyclictest437-21ssh21:38:123
2450199118,2cyclictest29731-21head20:25:233
2450199118,2cyclictest29058-21ssh22:52:413
2450199118,2cyclictest28414-21ssh21:15:373
2450199118,2cyclictest24085-21ssh23:09:173
2450199118,2cyclictest24027-21grep00:25:243
2450199118,2cyclictest17766-21head22:45:283
2450199118,2cyclictest17083-21ssh00:21:083
2450199118,2cyclictest1607-21sh22:36:063
2450199118,2cyclictest15944-21head21:05:183
2450199118,2cyclictest14834-21grep22:05:283
2450199118,2cyclictest14772-21ssh21:27:093
2450199118,2cyclictest14561-21ssh23:03:263
2450199118,2cyclictest13439-21ssh00:00:193
2450199118,2cyclictest13222-21grep19:50:263
2450199117,3cyclictest7098-21ssh21:41:463
2450199117,1cyclictest18699-21cat20:05:003
2450199114,1cyclictest225-21systemd-journal20:40:153
2450199110,11cyclictest0-21swapper/323:35:203
2449999119,1cyclictest8576-21df20:50:162
2449999119,1cyclictest4903-21sed20:40:272
2449999119,1cyclictest10028-21ls21:25:002
2449999118,2cyclictest9802-21sed19:45:172
2449999118,2cyclictest29350-21date19:20:002
2449999118,2cyclictest24989-21ssh22:11:332
2449999118,2cyclictest21464-21cut20:10:162
2449999118,2cyclictest18515-21dpkg20:05:002
2449999118,2cyclictest18396-21sh23:25:132
2449999112,6cyclictest0-21swapper/219:24:442
2449999110,10cyclictest31239-21ssh23:51:142
2449999110,10cyclictest3054-21cat20:40:002
2449999110,10cyclictest11748-21ssh00:18:312
2449499118,2cyclictest8987-21grep19:45:001
2449499118,2cyclictest8977-21ssh22:21:131
2449499118,2cyclictest8977-21ssh22:21:121
2449499118,2cyclictest636-21head00:30:281
2449499118,2cyclictest4747-21grep19:35:151
2449499118,2cyclictest3121-21ssh22:37:151
2449499118,2cyclictest29388-21ssh21:16:231
2449499118,2cyclictest25534-21ssh23:10:121
2449499118,2cyclictest25462-21grep19:10:161
2449499118,2cyclictest20119-21ssh22:28:201
2449499118,2cyclictest18562-21ssh22:46:031
2449499118,2cyclictest14708-21tail19:55:191
2449099119,1cyclictest7899-21grep22:20:270
2449099119,1cyclictest7899-21grep22:20:270
2449099119,1cyclictest3034-21tr23:15:230
2449099119,1cyclictest21599-21tr22:10:140
2449099118,2cyclictest6612-21readlink19:40:000
2449099118,2cyclictest3681-21ssh21:59:440
2449099118,2cyclictest31779-21cat22:15:270
2449099118,2cyclictest25248-21ssh00:25:560
2449099118,2cyclictest24763-21tr19:10:140
2449099118,2cyclictest18006-21head23:05:180
2449099118,2cyclictest12534-21basename21:00:010
2449099115,5cyclictest3263-21kernelversion21:20:190
2449099110,1cyclictest0-21swapper/023:35:040
2450199108,1cyclictest26338-21cat20:20:163
2450199108,1cyclictest11979-21cat20:55:273
2450199107,2cyclictest9746-21grep20:50:283
2450199107,2cyclictest5795-21ssh00:15:143
2450199107,2cyclictest25394-21cut19:10:173
2449999108,2cyclictest413-21tail19:25:172
2449999108,1cyclictest5213-21grep19:35:162
2449999108,1cyclictest26746-21tune2fs20:20:182
2449999108,1cyclictest19892-21grep21:30:172
2449999108,1cyclictest14098-21grep19:55:162
2449999108,1cyclictest12060-21cat19:50:152
2449999107,2cyclictest26830-21sh19:15:002
2449999107,2cyclictest24666-21grep20:15:192
2449999107,2cyclictest16794-21grep21:05:272
2449999107,2cyclictest13714-21irqstats21:00:202
2449499108,1cyclictest8513-21sh20:50:171
2449499108,1cyclictest2281-21cat20:35:271
2449499108,1cyclictest21307-21taskset20:10:041
2449499108,1cyclictest11615-21grep20:55:231
2449499107,2cyclictest9516-21tr19:45:161
2449499107,2cyclictest6383-21grep20:45:171
2449499107,2cyclictest30282-21taskset20:25:581
2449499107,2cyclictest29785-21cut19:20:141
2449499107,2cyclictest2817-21expr19:30:171
2449499107,2cyclictest23679-21chrt20:15:031
2449499107,2cyclictest19285-21seq20:05:161
2449099108,2cyclictest9422-21ssh00:35:450
2449099108,2cyclictest17482-21ssh23:43:350
2449099108,1cyclictest3803-21sed19:30:260
2449099108,1cyclictest296752sleep019:20:040
2449099108,1cyclictest18196-21ssh23:25:040
2449099108,1cyclictest134002sleep019:50:570
2449099107,2cyclictest23235-21taskset20:12:170
2449099105,2cyclictest16388-21apache_volume20:00:140
2449099104,5cyclictest16089-21cron20:00:000
2449099102,6cyclictest0-21swapper/019:45:150
245019996,2cyclictest6524-21cat20:45:183
244999997,1cyclictest2158-21cut19:30:142
244999996,2cyclictest19113-21grep20:05:152
244999991,2cyclictest225-21systemd-journal20:45:162
244949994,2cyclictest225-21systemd-journal20:00:071
244909998,1cyclictest28024-21taskset20:22:260
244909992,4cyclictest32704-21fschecks_count19:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional