You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat May 23, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1995121120,0sleep20-21swapper/212:11:202
1144129682,10sleep00-21swapper/007:06:320
1144829380,9sleep20-21swapper/207:06:392
1147328665,6sleep10-21swapper/107:06:571
1144428370,9sleep30-21swapper/307:06:363
132292730,0sleep20-21swapper/211:48:402
11787992926,2cyclictest13-21ksoftirqd/008:25:000
1179299280,28cyclictest0-21swapper/212:16:112
1178899244,5cyclictest2852-21fschecks_count10:25:171
1178799240,0cyclictest0-21swapper/007:20:150
11787992319,3cyclictest13-21ksoftirqd/008:15:010
11797992222,0cyclictest0-21swapper/312:19:123
1179799220,22cyclictest0-21swapper/310:40:183
1179299220,22cyclictest0-21swapper/211:22:532
1178899220,2cyclictest0-21swapper/110:08:081
11787992221,1cyclictest131ksoftirqd/010:43:560
11787992220,1cyclictest13-21ksoftirqd/011:58:500
11787992219,2cyclictest18987-21ssh10:35:120
11787992217,2cyclictest13-21ksoftirqd/012:35:060
1179799211,5cyclictest0-21swapper/309:45:133
1178899213,16cyclictest4822-21diskmemload11:40:541
1179799200,18cyclictest0-21swapper/312:31:543
1179299200,18cyclictest0-21swapper/210:50:452
11787992018,1cyclictest161rcuc/009:40:130
11787992017,1cyclictest13-21ksoftirqd/011:05:020
1179799194,10cyclictest13947-21runrttasks11:10:283
1179799190,1cyclictest0-21swapper/309:27:293
11788991918,1cyclictest0-21swapper/111:51:501
1178899190,17cyclictest0-21swapper/112:15:121
1179799186,3cyclictest225-21systemd-journal09:32:093
1179799180,18cyclictest0-21swapper/312:12:113
11788991817,1cyclictest0-21swapper/110:46:411
1178799189,3cyclictest13-21ksoftirqd/009:28:580
11787991817,1cyclictest0-21swapper/011:00:520
1178799181,2cyclictest141rcu_preempt11:22:230
1178799180,1cyclictest141rcu_preempt09:49:180
11797991714,1cyclictest0-21swapper/308:38:043
1179299170,1cyclictest0-21swapper/211:09:002
1179299170,1cyclictest0-21swapper/209:35:192
1179299170,16cyclictest0-21swapper/210:31:212
1178899172,14cyclictest4822-21diskmemload10:50:411
1178899171,2cyclictest17772-21sed12:30:001
1178899170,17cyclictest0-21swapper/109:52:241
1178899170,16cyclictest0-21swapper/109:07:091
1178899170,16cyclictest0-21swapper/107:45:011
1178799178,8cyclictest8732-21sensors11:45:220
11787991714,2cyclictest13-21ksoftirqd/012:03:090
11787991711,3cyclictest13-21ksoftirqd/010:10:320
11797991614,1cyclictest0-21swapper/312:29:183
11797991614,1cyclictest0-21swapper/311:50:263
1179799160,1cyclictest0-21swapper/310:58:103
1179799160,14cyclictest0-21swapper/310:51:323
1178899160,1cyclictest4822-21diskmemload11:26:341
1178799167,2cyclictest13-21ksoftirqd/011:44:160
1178799163,1cyclictest141rcu_preempt09:12:060
1178799162,7cyclictest6251-21basename08:05:010
11787991615,1cyclictest161rcuc/010:54:590
11787991615,1cyclictest0-21swapper/010:29:490
1178799160,13cyclictest13-21ksoftirqd/010:58:440
1179799152,2cyclictest141rcu_preempt12:38:273
11797991515,0cyclictest0-21swapper/311:09:413
11797991513,1cyclictest0-21swapper/310:06:173
11797991511,3cyclictest4822-21diskmemload09:42:343
1179299154,6cyclictest21570-21perf07:29:592
11788991510,0cyclictest141rcu_preempt09:30:131
1178899150,14cyclictest0-21swapper/109:45:341
1178799154,6cyclictest21574-21munin-run07:30:000
1178799151,5cyclictest13-21ksoftirqd/010:15:170
11787991513,2cyclictest161rcuc/009:50:120
11787991510,5cyclictest13-21ksoftirqd/012:15:220
1178799150,1cyclictest0-21swapper/010:09:550
1179799142,1cyclictest21470-21apache_accesses11:15:123
1179799142,1cyclictest21470-21apache_accesses11:15:113
1179799141,1cyclictest141rcu_preempt12:23:173
11797991411,2cyclictest24731-21grep11:55:183
11797991411,2cyclictest18681-21grep08:30:153
1179799140,14cyclictest0-21swapper/310:18:033
11792991411,2cyclictest3475-21head09:05:172
1178899141,1cyclictest4822-21diskmemload10:41:081
11788991411,2cyclictest23066-21grep09:20:171
11788991410,2cyclictest5477-21grep09:10:151
1178899140,4cyclictest4822-21diskmemload10:55:381
1178899140,2cyclictest0-21swapper/112:08:011
1178899140,14cyclictest0-21swapper/112:02:051
1178799149,2cyclictest13-21ksoftirqd/011:28:070
1178799144,1cyclictest141rcu_preempt10:00:130
11787991414,0cyclictest0-21swapper/008:35:160
11787991413,1cyclictest13-21ksoftirqd/011:30:020
1178799141,1cyclictest141rcu_preempt11:13:560
11787991411,1cyclictest13-21ksoftirqd/009:58:450
1178799140,2cyclictest0-21swapper/011:36:190
1179799131,1cyclictest0-21swapper/311:23:333
11797991311,1cyclictest198542sleep309:18:343
11797991311,1cyclictest0-21swapper/311:00:583
11797991310,2cyclictest18783-21grep07:20:273
11797991310,2cyclictest15339-21grep08:20:253
1179799130,13cyclictest0-21swapper/310:48:453
1179799130,12cyclictest552-21grep07:50:213
1179799130,11cyclictest23278-21ssh10:37:143
1179299139,3cyclictest1186-21ntpq12:00:202
1179299136,6cyclictest26890-21munin-run10:39:592
11792991310,2cyclictest9268-21cron12:25:002
11792991310,2cyclictest8872-21ssh10:29:082
11792991310,2cyclictest7575-21cat11:25:262
11792991310,2cyclictest3170-21cat07:55:262
11792991310,2cyclictest31447-21tail09:25:182
11792991310,2cyclictest29876-21chrt09:24:512
11792991310,2cyclictest28698-21cron07:44:592
11792991310,2cyclictest2489-21cat10:05:262
11792991310,2cyclictest21688-21grep10:55:222
11792991310,2cyclictest14380-21grep08:20:162
1179299130,13cyclictest0-21swapper/212:07:322
1179299130,0cyclictest0-21swapper/209:42:562
1178899139,3cyclictest27334-21head07:40:171
1178899139,3cyclictest18452-21basename10:35:001
1178899139,2cyclictest0-21swapper/107:10:131
1178899134,8cyclictest0-21swapper/107:25:151
1178899133,5cyclictest225-21systemd-journal10:14:591
11788991313,0cyclictest0-21swapper/110:23:381
11788991310,2cyclictest7201-21sh11:45:131
11788991310,2cyclictest4979-21tr11:05:151
1178899130,13cyclictest0-21swapper/109:55:121
1178899130,13cyclictest0-21swapper/109:16:341
1178799139,3cyclictest29294-21wc08:50:210
1178799139,3cyclictest22566-21tail07:30:170
1178799139,2cyclictest13-21ksoftirqd/009:16:580
1178799139,1cyclictest161rcuc/007:55:000
1178799138,3cyclictest161rcuc/009:00:260
1178799137,4cyclictest13-21ksoftirqd/011:51:110
1178799135,3cyclictest141rcu_preempt12:05:560
1178799132,9cyclictest8601-21cron08:10:000
1178799132,4cyclictest13-21ksoftirqd/010:20:130
1178799132,2cyclictest13-21ksoftirqd/007:15:110
11787991311,1cyclictest0-21swapper/010:45:150
11787991310,2cyclictest31972-21grep08:55:280
11787991310,2cyclictest2196-21head12:20:160
11787991310,2cyclictest19008-21grep12:30:160
1178799130,13cyclictest0-21swapper/011:16:560
1178799130,13cyclictest0-21swapper/011:16:560
1179799129,2cyclictest8875-21cat11:45:253
1179799129,2cyclictest8633-21cat08:10:013
1179799129,2cyclictest7132-21grep11:25:213
1179799129,2cyclictest5479-21tr09:10:143
1179799129,2cyclictest28209-21grep07:40:273
1179799129,2cyclictest22593-21head07:30:173
1179799129,2cyclictest2218-21grep07:55:163
1179799129,2cyclictest1710-21cat12:00:263
1179799128,3cyclictest29468-21tr07:45:153
1179799128,3cyclictest25483-21cat08:45:003
1179799128,3cyclictest13365-21munin-run08:20:013
1179799128,2cyclictest10868-21tr10:30:163
1179799127,5cyclictest1035-21packagekitd09:38:133
1179799122,6cyclictest16552-21awk07:20:003
1179799122,5cyclictest0-21swapper/311:30:143
11797991210,2cyclictest16691-21apache209:56:233
1179799120,1cyclictest0-21swapper/312:05:153
1179299129,2cyclictest3708-21tr10:45:152
1179299129,2cyclictest30812-21grep07:45:272
1179299129,2cyclictest26334-21grep08:45:172
1179299129,2cyclictest24900-21cut07:35:312
1179299129,2cyclictest23366-21grep11:15:262
1179299129,2cyclictest23366-21grep11:15:262
1179299129,2cyclictest2125-21ssh12:40:002
1179299129,2cyclictest17708-21ssh12:30:002
1179299129,2cyclictest15537-21grep11:30:192
1179299129,2cyclictest13731-21sed07:10:262
11792991210,1cyclictest11025-21sed10:10:262
1179299120,12cyclictest0-21swapper/210:16:542
1178899129,2cyclictest31648-21ssh11:20:251
1178899129,2cyclictest31474-21grep09:25:171
1178899129,2cyclictest31440-21tail08:55:211
1178899129,2cyclictest23430-21chrt09:40:121
1178899129,2cyclictest19663-21wc12:30:211
1178899129,2cyclictest19186-21sh07:25:001
1178899129,2cyclictest18062-21grep10:15:161
1178899129,2cyclictest17987-21uname12:10:181
1178899129,2cyclictest15329-21head11:10:401
1178899128,1cyclictest1297-21ssh12:20:091
1178899127,4cyclictest26529-21timerandwakeup10:00:261
1178899123,4cyclictest21434-21/usr/sbin/munin11:15:141
1178899123,4cyclictest21434-21/usr/sbin/munin11:15:141
1178899121,9cyclictest0-21swapper/108:35:151
11788991210,1cyclictest25-21ksoftirqd/108:25:001
1178899120,4cyclictest0-21swapper/111:55:131
1178899120,11cyclictest3873-21sendmail-msp08:00:011
1178899120,11cyclictest15309-21tr09:35:151
1178799129,2cyclictest28171-21wc07:40:260
1178799129,2cyclictest27731-21sleep08:50:000
1178799129,2cyclictest25546-21wc07:35:370
1178799123,8cyclictest13-21ksoftirqd/009:32:090
1178799121,1cyclictest141rcu_preempt09:37:190
1179799118,2cyclictest9169-21grep10:10:153
1179799118,2cyclictest5310-21ssh10:26:223
1179799118,2cyclictest4951-21basename08:00:193
1179799118,2cyclictest4881-21cat09:10:003
1179799118,2cyclictest39-21ksoftirqd/309:00:133
1179799118,2cyclictest28750-21ssh11:38:323
1179799118,2cyclictest28424-21ssh09:23:423
1179799118,2cyclictest26520-21grep10:20:163
1179799118,2cyclictest26138-21chrt07:37:573
1179799118,2cyclictest2353-21ssh11:41:403
1179799110,10cyclictest24726-21ssh10:00:133
1179299119,2cyclictest0-21swapper/211:58:252
1179299119,1cyclictest5360-21seq09:10:142
1179299119,1cyclictest4720-21grep08:00:172
1179299119,1cyclictest32483-21basename09:00:002
1179299118,2cyclictest9653-21grep09:50:272
1179299118,2cyclictest9342-21cat08:10:162
1179299118,2cyclictest7800-21rm09:30:192
1179299118,2cyclictest7306-21cat08:05:192
1179299118,2cyclictest545-21tr09:00:132
1179299118,2cyclictest31908-21grep11:40:172
1179299118,2cyclictest31518-21sh11:01:052
1179299118,2cyclictest30535-21ssh10:22:162
1179299118,2cyclictest27159-21ssh10:40:022
1179299118,2cyclictest25253-21expr10:00:162
1179299118,2cyclictest23927-21cron07:35:002
1179299118,2cyclictest23643-21head11:35:182
1179299118,2cyclictest2112-21ssh09:46:012
1179299118,2cyclictest21071-21grep08:35:172
1179299118,2cyclictest18716-21expr12:30:142
1179299118,2cyclictest18682-21tr08:30:152
1179299118,2cyclictest17990-21sh09:55:252
1179299118,2cyclictest17643-21ssh09:16:502
1179299118,2cyclictest13327-21tr11:10:142
1179299117,3cyclictest19189-21turbostat.cron07:25:002
1179299110,10cyclictest16697-21missed_timers11:50:202
1178899119,1cyclictest1526-21cat07:55:011
1178899118,2cyclictest31893-21ssh12:37:581
1178899118,2cyclictest26307-21sh07:40:001
1178899118,2cyclictest23453-21grep07:30:261
1178899118,2cyclictest22627-21ssh11:35:071
1178899118,2cyclictest17173-21ssh11:31:031
1178899118,2cyclictest16691-21apache208:50:131
1178899118,2cyclictest15964-21cat07:15:251
1178899118,2cyclictest10522-21grep08:10:281
1178899117,3cyclictest16303-21tr08:25:151
1178899114,2cyclictest221irq_work/111:04:311
1178799118,2cyclictest19179-21grep08:30:160
1178799118,2cyclictest10771-21sh12:25:180
1178799117,3cyclictest947-21sed07:50:260
1178799116,1cyclictest13-21ksoftirqd/012:10:470
1178799115,2cyclictest141rcu_preempt10:30:290
1178799113,4cyclictest225-21systemd-journal07:15:000
1178799111,4cyclictest25357-21awk08:44:590
1179799108,1cyclictest7713-21ssh09:50:123
1179799108,1cyclictest30337-21cat08:55:133
1179799108,1cyclictest273892sleep308:46:013
1179799108,1cyclictest20073-21tune2fs07:25:163
1179799108,1cyclictest12239-21cat07:10:153
1179299108,1cyclictest12215-21ls08:15:182
1179299107,2cyclictest28751-21grep08:50:172
1179299107,2cyclictest24131-21cat08:40:192
1179299107,2cyclictest16730-21tune2fs08:25:162
1178899108,1cyclictest19035-21rm10:35:131
1178899108,1cyclictest1520-21sed09:00:201
1178899108,1cyclictest13187-21chrt08:17:531
1178899107,2cyclictest8504-21taskset08:08:521
1178899107,2cyclictest4082-21chrt08:00:071
1178899107,2cyclictest25707-21wget08:45:141
1178899107,2cyclictest25091-21taskset08:41:581
1178899107,2cyclictest18684-21cut08:30:151
1178799108,1cyclictest4060-21which09:05:220
1178799108,1cyclictest312062sleep007:49:040
1178799107,2cyclictest23997-21sed09:20:260
1178799107,2cyclictest12571-21wc08:15:220
117979997,1cyclictest28360-21cat08:50:153
117979996,2cyclictest9988-21nfs4_client08:10:223
117979992,3cyclictest225-21systemd-journal08:28:583
117929998,1cyclictest474-21ntpq07:50:212
117929998,1cyclictest15138-21expr07:15:162
117879994,2cyclictest13-21ksoftirqd/008:30:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional