You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 17:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jun 18, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2566921070,1sleep025661-21sensors10:25:240
78829684,8sleep00-21swapper/007:06:090
86229077,9sleep20-21swapper/207:07:072
3193528874,9sleep30-21swapper/307:05:163
102828265,13sleep10-21swapper/107:09:151
119199250,2cyclictest0-21swapper/310:00:403
183792240,0sleep20-21swapper/212:16:262
176132230,9sleep013-21ksoftirqd/008:50:130
118699230,23cyclictest0-21swapper/211:01:372
118299230,1cyclictest0-21swapper/111:11:221
118099233,8cyclictest0-21swapper/012:30:120
119199221,20cyclictest0-21swapper/311:13:013
119199220,22cyclictest0-21swapper/311:50:093
118699221,16cyclictest0-21swapper/208:05:142
118299220,22cyclictest0-21swapper/111:29:561
1191992117,3cyclictest26877-21diskmemload10:50:243
119199210,20cyclictest0-21swapper/309:44:063
118299217,8cyclictest0-21swapper/112:00:121
1182992119,1cyclictest0-21swapper/107:40:131
118099210,20cyclictest0-21swapper/008:55:120
118699201,1cyclictest21489-21apt-get09:45:112
118699200,17cyclictest0-21swapper/210:13:362
118299191,18cyclictest30432-21kworker/u8:110:29:041
118299190,18cyclictest0-21swapper/111:35:261
69652180,0sleep00-21swapper/010:53:010
1191991817,1cyclictest0-21swapper/310:13:023
118699180,1cyclictest0-21swapper/210:38:422
118299180,17cyclictest0-21swapper/110:13:371
118099180,2cyclictest0-21swapper/010:31:050
118099180,16cyclictest0-21swapper/011:25:140
119199172,15cyclictest0-21swapper/311:38:503
1191991717,0cyclictest0-21swapper/312:12:403
1186991717,0cyclictest0-21swapper/210:31:012
118699170,2cyclictest0-21swapper/212:35:342
1182991715,1cyclictest16505-21sed08:45:241
1180991716,1cyclictest0-21swapper/012:23:270
1180991712,3cyclictest28015-21diskstats09:10:150
118099170,1cyclictest0-21swapper/011:47:150
119199160,15cyclictest39-21ksoftirqd/309:10:533
118699168,4cyclictest225-21systemd-journal08:23:112
1186991615,1cyclictest32-21ksoftirqd/212:05:162
1186991611,3cyclictest5900-21cron07:20:002
118299160,16cyclictest0-21swapper/111:51:321
1180991615,1cyclictest0-21swapper/010:01:320
118099160,2cyclictest141rcu_preempt09:34:070
1191991515,0cyclictest0-21swapper/310:25:023
1191991511,4cyclictest0-21swapper/312:36:563
119199150,14cyclictest0-21swapper/309:52:463
119199150,0cyclictest0-21swapper/312:33:243
1182991512,1cyclictest26877-21diskmemload11:17:561
118299150,1cyclictest0-21swapper/111:31:001
118099151,1cyclictest0-21swapper/012:01:330
1180991511,3cyclictest15109-21cron10:20:000
118099150,14cyclictest0-21swapper/011:51:410
1191991413,1cyclictest1174-21cyclictest10:15:323
1191991411,2cyclictest1112-21ls10:30:173
119199141,10cyclictest18383-21/usr/sbin/munin11:00:143
1191991410,3cyclictest8159-21chrt07:22:353
119199140,1cyclictest0-21swapper/309:49:033
118699148,5cyclictest12006-21/usr/sbin/munin09:20:222
1186991413,1cyclictest0-21swapper/209:30:022
1186991410,3cyclictest16091-21cron12:35:002
1182991413,1cyclictest55550irq/126-enp2s0-11:03:261
118299140,0cyclictest0-21swapper/109:10:091
118099141,3cyclictest141rcu_preempt10:11:000
1180991411,2cyclictest11536-21munin-run09:20:000
1180991410,3cyclictest22352-21grep07:50:260
119199139,3cyclictest3589-21head08:20:163
119199139,2cyclictest14312-21head07:35:173
119199133,5cyclictest0-21swapper/311:45:203
1191991310,2cyclictest29211-21readlink08:05:223
1191991310,2cyclictest20580-21ssh09:25:133
1191991310,2cyclictest10786-21sed08:35:153
119199130,12cyclictest4414-21tr11:30:153
119199130,12cyclictest16217-21wc12:15:203
118699136,6cyclictest0-21swapper/210:25:182
1186991310,2cyclictest9277-21grep08:30:242
1186991310,2cyclictest32-21ksoftirqd/209:44:052
1186991310,2cyclictest31748-21wc12:25:002
1186991310,2cyclictest31723-21cat09:50:262
1186991310,2cyclictest28347-21sleep09:30:002
1186991310,2cyclictest27120-21grep08:00:252
1186991310,2cyclictest24550-21cat07:55:252
118699130,13cyclictest0-21swapper/212:14:562
118699130,12cyclictest25360-21grep09:05:152
118699130,12cyclictest0-21swapper/210:55:192
118299139,3cyclictest30926-21uniq08:10:161
118299139,3cyclictest21249-21chrt08:55:211
118299139,2cyclictest14021-21sed09:40:171
118299139,2cyclictest10317-21wc08:35:141
118299132,7cyclictest225-21systemd-journal07:35:001
1182991313,0cyclictest0-21swapper/110:07:191
118299131,10cyclictest0-21swapper/110:40:141
1182991310,2cyclictest7996-21grep10:35:001
1182991310,2cyclictest31766-21wc12:05:201
1182991310,2cyclictest3124-21head07:10:241
1182991310,2cyclictest30060-21tail09:30:201
1182991310,2cyclictest26877-21diskmemload09:35:131
1182991310,2cyclictest22395-21wc11:40:211
1182991310,2cyclictest22395-21wc11:40:211
1182991310,2cyclictest15071-21wc07:35:261
118299130,1cyclictest0-21swapper/112:28:061
118299130,12cyclictest0-21swapper/110:24:221
118099139,3cyclictest19208-21sort11:00:160
118099139,3cyclictest17007-21grep12:15:270
118099134,7cyclictest15083-21wget08:45:130
118099132,6cyclictest17929-21sendmail11:00:000
1180991310,2cyclictest8617-21grep12:10:280
1180991310,2cyclictest4533-21grep08:20:250
1180991310,2cyclictest11210-21cut08:35:200
118099130,7cyclictest32380-21basename08:15:000
118099130,1cyclictest0-21swapper/009:50:130
118099130,13cyclictest0-21swapper/010:10:000
118099130,12cyclictest0-21swapper/008:00:120
119199129,2cyclictest4988-21grep09:35:143
119199129,2cyclictest4552-21grep07:15:163
119199129,2cyclictest24564-21cat09:05:003
119199129,2cyclictest11553-21tr11:15:133
119199128,3cyclictest7375-21basename08:30:003
119199128,3cyclictest7375-21basename08:30:003
119199128,2cyclictest32530-21grep12:05:273
119199128,2cyclictest28214-21tr11:25:133
119199124,5cyclictest11526-21systemctl09:20:003
119199121,8cyclictest0-21swapper/310:06:433
119199120,11cyclictest0-21swapper/311:55:113
118699129,3cyclictest0-21swapper/210:52:442
118699129,2cyclictest616-21chrt09:13:172
118699129,2cyclictest5162-21grep11:30:192
118699129,2cyclictest4441-21smartctl09:15:172
118699129,2cyclictest29342-21sshd11:45:112
118699129,2cyclictest22832-21cron07:55:002
118699129,2cyclictest20042-21sed07:45:252
118699129,2cyclictest15748-21head08:45:162
118699128,3cyclictest12108-21apt-get08:40:022
1186991210,1cyclictest0-21swapper/212:04:412
118699120,2cyclictest0-21swapper/212:29:102
118699120,1cyclictest0-21swapper/210:24:222
118699120,11cyclictest0-21swapper/211:43:432
118699120,11cyclictest0-21swapper/211:43:432
118299129,2cyclictest8397-21sed08:30:151
118299129,2cyclictest6498-21grep07:20:131
118299129,2cyclictest30702-21grep09:50:171
118299129,2cyclictest2848-21basename08:20:131
118299129,2cyclictest27350-21head10:45:261
118299129,2cyclictest26169-21chrt08:00:151
118299129,2cyclictest25492-21grep09:05:161
118299129,2cyclictest209902sleep111:20:191
118299129,2cyclictest17810-21cut12:35:171
118299129,2cyclictest14414-21chrt10:19:281
118299129,2cyclictest10366-21sed10:35:261
118299128,3cyclictest899-21grep08:15:141
118299128,3cyclictest22177-21wc07:50:251
118299128,3cyclictest14878-21wc11:55:171
118299121,9cyclictest6647-21/usr/sbin/munin12:10:201
118299121,7cyclictest0-21swapper/110:52:441
1182991211,1cyclictest3138-21kworker/1:012:17:461
118299120,9cyclictest0-21swapper/109:20:131
118299120,1cyclictest0-21swapper/109:15:031
118099129,2cyclictest6961-21tail07:20:150
118099129,2cyclictest5443-21tune2fs09:35:160
118099129,2cyclictest2517-21grep07:10:180
118099129,2cyclictest23360-21tr07:55:140
118099129,2cyclictest22451-21grep09:00:110
118099129,2cyclictest21496-21grep11:40:150
118099129,2cyclictest21496-21grep11:40:150
118099129,2cyclictest20573-21wc07:50:000
118099129,2cyclictest14373-21grep07:35:180
118099129,2cyclictest12001-21cut07:30:160
118099121,7cyclictest29362-21cron09:50:000
118099120,7cyclictest7730-21basename12:30:000
118099120,7cyclictest0-21swapper/010:35:040
118099120,12cyclictest0-21swapper/010:44:030
119199118,2cyclictest7817-21egrep08:30:123
119199118,2cyclictest31517-21ssh09:31:023
119199118,2cyclictest28418-21ssh11:05:223
119199118,2cyclictest26200-21head10:45:173
119199118,2cyclictest25787-21missed_timers09:05:193
119199118,2cyclictest2515-21ssh12:26:023
119199118,2cyclictest23546-21rm12:20:053
119199118,2cyclictest20585-21sed11:20:153
119199118,2cyclictest17240-21chrt07:40:223
119199118,2cyclictest16722-21sh10:40:033
119199118,2cyclictest12096-21ssh09:20:153
119199118,2cyclictest11126-21cat10:55:193
119199118,2cyclictest10554-21taskset10:35:293
119199117,3cyclictest7897-21sed09:55:273
119199117,3cyclictest25203-21cron08:00:003
119199116,4cyclictest20924-21apache_accesses11:40:113
119199116,4cyclictest20924-21apache_accesses11:40:113
119199116,3cyclictest17617-21apt-get08:50:123
119199115,5cyclictest16865-21nfs4_client10:20:193
1191991110,1cyclictest26877-21diskmemload12:01:123
118699118,2cyclictest9078-21ssh09:56:192
118699118,2cyclictest6611-21date09:35:252
118699118,2cyclictest5654-21cat08:25:142
118699118,2cyclictest5654-21cat08:25:142
118699118,2cyclictest4413-21ssh11:10:222
118699118,2cyclictest3589-21sed07:15:002
118699118,2cyclictest31313-21chrt11:26:212
118699118,2cyclictest29477-21ssh11:05:562
118699118,2cyclictest27704-21ssh10:45:402
118699118,2cyclictest27317-21sh11:24:462
118699118,2cyclictest24139-21tr10:05:222
118699118,2cyclictest17257-21tr10:40:152
118699118,2cyclictest15119-21expr10:00:172
118699118,2cyclictest12316-21grep07:30:192
118699118,2cyclictest12196-21cat11:15:162
118699117,3cyclictest8977-21timerandwakeup10:15:252
118699116,2cyclictest225-21systemd-journal11:40:002
118699111,8cyclictest0-21swapper/211:55:112
118699110,10cyclictest727-21expr08:15:142
118299118,2cyclictest7393-21ssh09:55:221
118299118,2cyclictest5818-21cut08:25:161
118299118,2cyclictest5818-21cut08:25:161
118299118,2cyclictest29331-21ssh11:45:111
118299118,2cyclictest27598-21sort11:05:161
118299118,2cyclictest23825-21tail07:55:181
118299118,2cyclictest23120-21grep09:00:161
118299118,2cyclictest21713-21ssh09:25:191
118299118,2cyclictest21287-21ssh09:45:031
118299118,2cyclictest14268-21cat10:00:111
118299118,2cyclictest12757-21ssh10:55:591
118299118,2cyclictest10526-21head07:25:271
118299117,3cyclictest8191-21ssh12:30:111
118299117,2cyclictest17836-21grep08:50:131
118299110,11cyclictest0-21swapper/112:23:001
118099118,2cyclictest9113-21ssh11:32:440
118099118,2cyclictest7331-21taskset08:29:450
118099118,2cyclictest7331-21taskset08:29:450
118099118,2cyclictest5923-21cut09:55:140
118099118,2cyclictest3938-21sed11:10:180
118099118,2cyclictest30852-21ssh11:06:590
118099118,2cyclictest28977-21ssh10:46:400
118099118,2cyclictest25412-21sort09:05:150
118099118,2cyclictest21250-21cat11:20:210
118099118,2cyclictest20594-21grep09:25:130
118099118,2cyclictest19236-21ssh12:35:540
118099118,2cyclictest1697-21ssh12:06:350
118099118,2cyclictest14378-21sed09:40:200
118099118,2cyclictest13492-21ssh11:35:180
118099118,2cyclictest12894-21tail09:20:170
118099118,2cyclictest11218-21chrt11:15:090
118099113,5cyclictest8848-21apt-get07:25:140
118099110,10cyclictest19717-21ssh11:58:250
119199108,2cyclictest14569-21taskset08:43:143
119199108,1cyclictest2431-21taskset08:18:493
119199108,1cyclictest22680-21chrt07:53:083
119199108,1cyclictest203622sleep307:48:513
119199108,1cyclictest15273-21expr08:45:133
119199108,1cyclictest11173-21taskset07:30:013
119199107,2cyclictest3451-21chrt07:14:133
119199107,2cyclictest30989-21expr08:10:163
119199107,2cyclictest21982-21chrt08:58:393
118699108,1cyclictest5362-21basename11:50:122
118699108,1cyclictest24313-21taskset09:02:462
118699108,1cyclictest17569-21chrt08:50:012
118699108,1cyclictest10107-21uname07:25:232
118699107,2cyclictest6457-21cat07:20:142
118699107,2cyclictest31991-21chrt08:10:262
118699107,2cyclictest20318-21expr08:55:142
118699107,2cyclictest16773-21chrt07:40:172
118699107,2cyclictest13728-21wget07:35:142
118699107,2cyclictest10256-21grep08:35:132
118299108,1cyclictest5647-21taskset07:16:511
118299108,1cyclictest29665-21taskset08:06:031
118299108,1cyclictest13498-21cat08:40:171
118299107,2cyclictest19659-21cut07:45:211
118099108,1cyclictest29790-21taskset08:07:390
118099108,1cyclictest16919-21grep07:40:190
118099108,1cyclictest12909-21wc08:40:130
118099107,2cyclictest16834-21tr10:20:180
118099107,2cyclictest1184-21cut08:15:160
118099101,6cyclictest0-21swapper/007:15:130
118099100,9cyclictest9459-21taskset08:31:340
11919997,1cyclictest9676-21cat07:25:193
11919996,2cyclictest25973-21expr08:00:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional