You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 20:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Apr 28, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31144210061,34sleep00-21swapper/007:05:320
3122229850,43sleep10-21swapper/107:06:341
3144429146,40sleep30-21swapper/307:09:253
263292880,1sleep30-21swapper/308:05:143
3139628371,8sleep20-21swapper/207:08:482
27702310,0chrt2772-21grep11:50:212
31583992726,1cyclictest0-21swapper/211:26:432
3158399260,25cyclictest0-21swapper/209:15:572
3158899243,15cyclictest8183-21apt-get08:35:153
3158899231,22cyclictest25756-21kworker/3:010:18:333
3158899231,22cyclictest25756-21kworker/3:010:18:333
3158899230,22cyclictest25756-21kworker/3:011:15:033
31583992320,2cyclictest520-21gmain12:29:552
31578992317,2cyclictest13-21ksoftirqd/011:30:290
31588992220,1cyclictest520-21gmain12:19:393
31578992221,1cyclictest13-21ksoftirqd/010:53:030
31578992219,2cyclictest13-21ksoftirqd/011:44:400
31578992214,2cyclictest13-21ksoftirqd/009:31:140
3158899210,21cyclictest25756-21kworker/3:011:21:433
3158899210,21cyclictest25756-21kworker/3:010:01:233
31583992117,4cyclictest0-21swapper/209:59:042
3158399210,18cyclictest0-21swapper/212:06:482
31581992121,0cyclictest0-21swapper/109:10:421
31578992118,1cyclictest13-21ksoftirqd/011:14:590
31578992112,3cyclictest13-21ksoftirqd/009:45:480
31588992019,1cyclictest0-21swapper/309:50:133
3158899200,1cyclictest25756-21kworker/3:011:26:573
3158199200,19cyclictest0-21swapper/111:35:391
3157899209,1cyclictest141rcu_preempt09:43:230
31578992019,1cyclictest161rcuc/012:30:000
3157899201,3cyclictest141rcu_preempt11:37:380
3158199192,1cyclictest141rcu_preempt10:42:051
31581991918,1cyclictest0-21swapper/111:56:181
31581991917,1cyclictest0-21swapper/110:38:101
3158199190,19cyclictest0-21swapper/112:01:131
3158199190,19cyclictest0-21swapper/110:34:001
3158199190,18cyclictest0-21swapper/111:54:501
3157899196,3cyclictest141rcu_preempt10:04:160
3157899193,13cyclictest13-21ksoftirqd/009:25:140
3157899192,1cyclictest12546-21taskset08:43:250
3157899192,1cyclictest12546-21taskset08:43:250
31578991919,0cyclictest161rcuc/011:18:560
31578991918,1cyclictest0-21swapper/009:52:130
31578991917,1cyclictest161rcuc/010:35:140
3157899190,1cyclictest0-21swapper/008:50:150
31581991818,0cyclictest0-21swapper/112:35:591
31581991817,1cyclictest0-21swapper/108:05:141
3157899187,1cyclictest141rcu_preempt12:24:430
31578991815,3cyclictest13-21ksoftirqd/011:49:370
31578991814,3cyclictest13-21ksoftirqd/012:06:320
31578991810,5cyclictest13-21ksoftirqd/010:05:290
263472180,0sleep10-21swapper/112:05:111
3158399174,12cyclictest14343-21python08:45:262
3158399170,16cyclictest0-21swapper/211:38:122
31581991714,2cyclictest0-21swapper/111:50:001
3158199170,1cyclictest0-21swapper/112:25:491
3157899178,8cyclictest0-21swapper/009:15:010
3157899177,9cyclictest13-21ksoftirqd/008:55:140
3157899172,14cyclictest8905-21apt-get07:30:020
3158899162,6cyclictest0-21swapper/312:35:123
3158899160,13cyclictest0-21swapper/309:26:523
31581991614,1cyclictest520-21gmain12:22:431
3158199161,1cyclictest141rcu_preempt09:15:161
3158199161,14cyclictest21765-21expr07:55:141
3158899156,6cyclictest10048-21/usr/sbin/munin11:55:153
3158399150,12cyclictest0-21swapper/210:42:112
3158199152,7cyclictest24130-21unixbench_multi09:05:271
3158199152,13cyclictest0-21swapper/111:44:141
31581991514,1cyclictest221irq_work/111:33:461
3158199151,1cyclictest141rcu_preempt09:59:351
3158199150,1cyclictest0-21swapper/109:41:571
3158199150,1cyclictest0-21swapper/109:29:441
3157899155,1cyclictest141rcu_preempt09:21:250
3157899154,3cyclictest13-21ksoftirqd/010:45:290
3157899152,3cyclictest141rcu_preempt10:30:450
31578991513,2cyclictest13-21ksoftirqd/009:59:210
31578991513,1cyclictest520-21gmain12:03:590
3158399148,5cyclictest32-21ksoftirqd/208:35:122
3158399145,6cyclictest15086-21grep11:00:142
3158399142,3cyclictest14828-21ssh10:20:312
3158399142,12cyclictest0-21swapper/208:11:332
31583991410,2cyclictest21121-21grep10:25:132
31583991410,2cyclictest15390-21ls11:20:012
3158399140,14cyclictest0-21swapper/209:46:222
3158199144,8cyclictest16003-21grep11:20:151
3158199143,4cyclictest141rcu_preempt10:47:051
31581991413,1cyclictest241rcuc/112:34:441
3158199140,8cyclictest0-21swapper/108:20:141
3157899149,4cyclictest13-21ksoftirqd/011:29:340
3157899148,5cyclictest13-21ksoftirqd/010:16:560
3157899148,5cyclictest13-21ksoftirqd/010:16:550
3157899143,10cyclictest13-21ksoftirqd/012:35:060
3157899142,2cyclictest13-21ksoftirqd/010:44:240
31578991411,2cyclictest13-21ksoftirqd/009:18:390
31578991410,4cyclictest13-21ksoftirqd/007:26:340
31578991410,3cyclictest31702-21grep08:15:160
3158899139,3cyclictest30778-21tr10:30:203
3158899139,3cyclictest2146-21grep08:20:203
3158899139,3cyclictest16506-21lockfile-remove11:40:003
3158899139,3cyclictest14377-21grep07:40:143
3158899139,2cyclictest21759-21tune2fs10:25:163
3158899132,1cyclictest24586-21diskmemload11:40:143
31588991312,1cyclictest198-21jbd2/sda2-808:50:183
31588991310,2cyclictest32699-21grep08:15:253
31588991310,2cyclictest25910-21sed09:10:163
31588991310,2cyclictest23067-21grep09:05:173
31588991310,2cyclictest15650-21sed11:00:173
31588991310,2cyclictest10487-21head09:20:163
3158899130,12cyclictest0-21swapper/312:05:143
3158399134,8cyclictest0-21swapper/208:55:192
3158399133,6cyclictest141rcu_preempt10:20:002
3158399133,6cyclictest141rcu_preempt10:20:002
3158399131,9cyclictest24591-21awk09:10:002
31583991310,3cyclictest32-21ksoftirqd/212:24:482
31583991310,2cyclictest3828-21grep12:30:152
3158199139,3cyclictest5836-21grep07:20:261
3158199138,4cyclictest21485-21ssh11:04:041
3158199137,4cyclictest25-21ksoftirqd/109:46:491
3158199131,9cyclictest141rcu_preempt09:36:361
31581991310,2cyclictest27176-21sed09:30:181
3158199130,10cyclictest141rcu_preempt10:28:321
3157899139,3cyclictest13-21ksoftirqd/012:30:150
3157899132,2cyclictest141rcu_preempt10:28:390
3157899131,2cyclictest141rcu_preempt10:21:180
3157899131,1cyclictest141rcu_preempt11:51:360
31578991310,3cyclictest13-21ksoftirqd/007:21:350
31578991310,2cyclictest15072-21grep11:00:140
31578991310,2cyclictest14282-21perf11:00:000
31578991310,2cyclictest12258-21ls12:15:190
3157899130,2cyclictest141rcu_preempt08:25:120
3157899130,12cyclictest1803-21grep07:15:120
3158899129,2cyclictest24634-21sed08:00:173
3158899129,2cyclictest18926-21head12:00:183
3158899128,3cyclictest28867-21wc12:25:193
3158899127,2cyclictest21563-21munin-run10:45:003
3158899123,3cyclictest19965-21df_inode12:20:153
3158899121,7cyclictest0-21swapper/309:15:213
3158899120,7cyclictest0-21swapper/311:45:223
3158899120,2cyclictest25756-21kworker/3:011:13:533
3158899120,11cyclictest21009-21sh10:05:183
3158399129,2cyclictest7022-21dump-pmu-power11:15:002
3158399129,2cyclictest6452-21grep08:30:162
3158399129,2cyclictest3870-21cron07:20:002
3158399129,2cyclictest31802-21grep08:15:162
3158399129,2cyclictest2754-21rm09:55:002
3158399129,2cyclictest26732-21wc09:10:252
3158399129,2cyclictest19446-21dpkg10:05:012
3158399129,2cyclictest11636-21sh12:15:142
3158399128,3cyclictest19601-21ssh12:00:242
3158399128,3cyclictest12832-21grep12:35:182
3158399126,3cyclictest0-21swapper/210:54:472
3158399125,6cyclictest0-21swapper/210:57:582
31583991211,1cyclictest0-21swapper/211:21:192
3158399120,2cyclictest0-21swapper/211:40:122
3158399120,0cyclictest0-21swapper/210:46:322
3158199129,2cyclictest9273-21cat08:35:201
3158199129,2cyclictest29341-21ssh10:10:171
3158199129,2cyclictest28782-21awk09:50:231
3158199129,2cyclictest19895-21wc09:00:001
3158199129,2cyclictest14089-21head10:20:201
3158199129,2cyclictest14003-21sort07:40:011
3158199128,3cyclictest8375-21grep11:15:181
3158199128,3cyclictest19892-21grep07:50:191
3158199128,3cyclictest12183-21grep08:40:241
3158199128,3cyclictest12183-21grep08:40:241
31581991211,1cyclictest0-21swapper/111:26:191
3158199120,12cyclictest0-21swapper/112:17:301
3158199120,12cyclictest0-21swapper/110:01:231
3157899129,2cyclictest26299-21tr08:05:140
3157899129,2cyclictest13-21ksoftirqd/009:40:000
3157899128,3cyclictest3160-21cron08:25:000
3157899127,5cyclictest13-21ksoftirqd/011:22:410
3157899125,6cyclictest21237-21munin-run07:55:000
3157899122,9cyclictest23891-21apt-get08:00:130
31578991211,1cyclictest161rcuc/011:55:150
3158899118,2cyclictest9347-21ssh09:39:263
3158899118,2cyclictest8474-21ssh09:57:593
3158899118,2cyclictest6610-21ssh10:55:153
3158899118,2cyclictest6269-21tail10:35:183
3158899118,2cyclictest3682-21apache_processe12:30:133
3158899118,2cyclictest3326-21grep12:10:153
3158899118,2cyclictest32552-21head07:10:193
3158899118,2cyclictest32382-21ssh09:33:363
3158899118,2cyclictest31373-21head10:50:183
3158899118,2cyclictest28361-21ssh10:10:053
3158899118,2cyclictest2747-21sed07:15:173
3158899118,2cyclictest2676-21ssh11:30:453
3158899118,2cyclictest26033-21ssh11:06:063
3158899118,2cyclictest22673-21sed10:45:163
3158899118,2cyclictest19091-21egrep09:45:143
3158899118,2cyclictest18474-21cut08:55:183
3158899118,2cyclictest17031-21tr07:45:153
3158899118,2cyclictest14380-21sed10:20:243
3158899118,2cyclictest12282-21smartctl09:40:263
3158899110,2cyclictest0-21swapper/311:50:073
3158899110,10cyclictest13945-21cat08:45:193
3158399118,2cyclictest9995-21ssh10:37:292
3158399118,2cyclictest9723-21grep09:20:132
3158399118,2cyclictest4043-21cat09:35:272
3158399118,2cyclictest29936-21open_files10:10:222
3158399118,2cyclictest27576-21grep08:05:222
3158399118,2cyclictest26200-21sed11:45:162
3158399118,2cyclictest25791-21ssh10:08:162
3158399118,2cyclictest2543-21ssh11:30:382
3158399118,2cyclictest24023-21grep11:05:182
3158399118,2cyclictest23567-21chrt07:59:442
3158399118,2cyclictest19369-21tr07:50:162
3158399118,2cyclictest19227-21ssh09:25:202
3158399118,2cyclictest15882-21ssh09:43:012
3158399118,2cyclictest1564-21sh10:32:232
3158399118,2cyclictest13449-21ssh11:56:482
3158399118,2cyclictest10408-21ssh12:14:592
3158399117,3cyclictest17420-21cron08:55:002
3158399113,1cyclictest24586-21diskmemload09:34:152
3158399110,10cyclictest4139-21basename07:20:112
3158199119,1cyclictest241rcuc/107:15:231
3158199118,2cyclictest9264-21rm07:30:001
3158199118,2cyclictest6566-21wget10:55:141
3158199118,2cyclictest6348-21ssh12:11:471
3158199118,2cyclictest5045-21ssh10:54:191
3158199118,2cyclictest4756-21grep10:15:131
3158199118,2cyclictest4756-21grep10:15:131
3158199118,2cyclictest32672-21wc11:10:181
3158199118,2cyclictest27179-21ssh10:09:211
3158199118,2cyclictest23457-21cpuspeed_turbos11:05:151
3158199118,2cyclictest21763-21tail09:00:271
3158199118,2cyclictest15539-21wc08:50:161
3158199118,2cyclictest14862-21grep07:40:161
3158199118,2cyclictest10463-21sed09:20:161
3157899118,2cyclictest6304-21ssh12:11:450
3157899118,2cyclictest25755-21ssh11:05:530
3157899118,2cyclictest20767-21grep09:00:170
3157899118,2cyclictest15374-21grep07:40:210
3157899118,2cyclictest15060-21ls08:50:000
3157899110,1cyclictest141rcu_preempt08:30:190
3158899108,1cyclictest4409-21expr07:20:163
3158899108,1cyclictest28600-21chrt08:10:073
3158899108,1cyclictest22984-21cat07:55:253
3158899108,1cyclictest21047-21wc09:00:193
3158899107,2cyclictest6508-21grep08:30:163
3158899107,2cyclictest3439-21chrt08:25:133
3158899107,2cyclictest20563-21wc07:50:273
3158899107,2cyclictest12060-21ls07:35:153
3158899107,2cyclictest11061-21df08:40:143
3158899107,2cyclictest11061-21df08:40:143
3158399108,1cyclictest9222-21touch07:30:002
3158399108,1cyclictest7840-21uname07:25:222
3158399108,1cyclictest32561-21sed07:10:202
3158399108,1cyclictest2933-21taskset08:22:102
3158399108,1cyclictest21201-21cut09:00:202
3158399107,2cyclictest14378-21tr07:40:142
3158399106,3cyclictest5115-21grep08:25:262
3158399103,6cyclictest0-21swapper/207:45:192
3158399100,9cyclictest0-21swapper/208:40:252
3158399100,9cyclictest0-21swapper/208:40:252
3158199108,1cyclictest29857-21sh08:10:201
3158199108,1cyclictest24217-21expr08:00:151
3158199108,1cyclictest15227-21cat08:50:001
3158199107,2cyclictest7423-21grep08:30:251
3158199107,2cyclictest6830-21tr07:25:151
3158199107,2cyclictest32306-21tr08:15:211
3158199107,2cyclictest31951-21grep07:10:171
3158199107,2cyclictest16715-21basename07:45:111
3158199100,9cyclictest4776-21basename08:25:231
3157899109,1cyclictest30514-21chrt08:11:240
3157899108,1cyclictest926-21ls07:10:280
3157899107,2cyclictest32206-21ssh10:11:420
3157899107,2cyclictest12829-21cat07:35:200
3157899101,4cyclictest225-21systemd-journal09:10:000
3157899100,9cyclictest8255-21aten2_r7power_p08:35:140
3157899100,1cyclictest141rcu_preempt07:55:110
315889997,1cyclictest8323-21taskset07:25:583
315889997,1cyclictest113862sleep307:33:453
315839997,1cyclictest13517-21chrt07:35:362
315839996,2cyclictest24554-21cut08:00:162
315789996,2cyclictest17132-21cat07:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional