You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 16:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu May 28, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1502329750,43sleep10-21swapper/107:05:061
1690429575,15sleep20-21swapper/207:06:442
1687029465,9sleep30-21swapper/307:06:173
1714529280,8sleep00-21swapper/007:09:510
115772480,0sleep20-21swapper/211:24:092
17243993028,1cyclictest0-21swapper/110:26:251
17243992626,0cyclictest0-21swapper/110:20:001
17243992626,0cyclictest0-21swapper/110:20:001
1725099240,0cyclictest0-21swapper/310:52:543
17240992421,2cyclictest94-21kswapd009:37:230
1725099230,0cyclictest0-21swapper/311:30:333
17243992319,3cyclictest0-21swapper/109:55:001
1724399230,1cyclictest0-21swapper/111:30:041
17240992320,2cyclictest10051-21diskmemload10:30:430
17246992219,2cyclictest0-21swapper/207:50:002
1724399226,9cyclictest0-21swapper/111:06:281
1724399220,0cyclictest0-21swapper/110:53:041
1724399216,15cyclictest10051-21diskmemload11:57:351
17243992119,2cyclictest10051-21diskmemload12:33:551
1724399211,7cyclictest0-21swapper/112:35:131
17243992117,4cyclictest0-21swapper/110:06:541
17243992117,3cyclictest0-21swapper/109:10:191
17243992015,5cyclictest0-21swapper/111:35:331
17243992012,7cyclictest0-21swapper/108:30:131
17243992011,8cyclictest10051-21diskmemload12:19:131
1724399200,20cyclictest0-21swapper/112:13:231
1725099193,2cyclictest3165-21apt-get08:55:133
1725099190,17cyclictest0-21swapper/310:22:103
1725099190,16cyclictest0-21swapper/310:16:243
1725099190,16cyclictest0-21swapper/310:16:243
1724399193,15cyclictest0-21swapper/110:01:281
17243991919,0cyclictest0-21swapper/110:58:451
17243991915,4cyclictest0-21swapper/109:35:221
17243991914,5cyclictest0-21swapper/110:11:051
1724399191,1cyclictest0-21swapper/110:36:091
1724399190,17cyclictest26299-21/usr/sbin/munin07:25:241
1724699180,1cyclictest0-21swapper/209:30:222
17243991814,2cyclictest0-21swapper/111:17:391
1724099186,6cyclictest2906-21munin-run08:55:010
17250991715,1cyclictest31390-21ssh10:00:153
1725099170,16cyclictest0-21swapper/309:14:443
1724699171,1cyclictest10051-21diskmemload10:28:252
1724699171,11cyclictest26971-21/usr/sbin/munin10:35:212
1724699170,16cyclictest0-21swapper/210:32:492
1724399178,5cyclictest10060-21grep08:00:171
17243991715,1cyclictest0-21swapper/109:55:391
1724099170,0cyclictest0-21swapper/011:50:380
1725099160,1cyclictest0-21swapper/312:03:463
1724699160,16cyclictest0-21swapper/212:07:292
1724399166,7cyclictest0-21swapper/109:26:051
17243991614,2cyclictest0-21swapper/109:18:491
1724399161,15cyclictest0-21swapper/111:54:421
1724399161,14cyclictest0-21swapper/109:43:011
17243991611,3cyclictest10051-21diskmemload12:20:401
1724399160,14cyclictest0-21swapper/111:28:291
1725099151,1cyclictest0-21swapper/312:29:563
1724699150,15cyclictest0-21swapper/212:36:162
1724699150,15cyclictest0-21swapper/211:39:182
1724399159,4cyclictest0-21swapper/107:35:171
1724399158,7cyclictest0-21swapper/110:21:591
1724399158,7cyclictest0-21swapper/109:47:101
1724399157,8cyclictest0-21swapper/111:02:081
1724399157,8cyclictest0-21swapper/107:45:251
1724399157,7cyclictest0-21swapper/109:30:261
1724399156,7cyclictest0-21swapper/110:49:271
1724399155,10cyclictest0-21swapper/109:23:201
1724399153,11cyclictest0-21swapper/111:48:261
17243991510,5cyclictest0-21swapper/110:42:091
17243991510,5cyclictest0-21swapper/110:30:051
1724099154,9cyclictest1024-21ntp_states11:55:190
1724099154,5cyclictest0-21swapper/011:40:000
1724099151,7cyclictest1042-21sendmail10:40:010
1724099150,9cyclictest20203-21awk09:35:000
1725099149,3cyclictest13514-21tr11:25:143
17250991411,2cyclictest26693-21cut08:35:163
17250991411,2cyclictest0-21swapper/312:10:263
17250991410,2cyclictest14779-21cut08:10:163
1724699145,6cyclictest2580-21cron12:35:012
1724699142,3cyclictest141rcu_preempt09:29:272
1724699141,5cyclictest19171-21ssh10:50:122
1724699140,4cyclictest0-21swapper/211:40:122
1724399149,4cyclictest17895-21awk08:15:241
1724399147,5cyclictest0-21swapper/111:22:451
1724399146,6cyclictest0-21swapper/111:40:451
1724399140,2cyclictest0-21swapper/112:25:411
1724399140,12cyclictest0-21swapper/111:14:371
1724099145,8cyclictest5280-21munin-run09:00:010
1724099143,6cyclictest1750-21basename07:45:000
17240991413,1cyclictest17232-21cyclictest07:40:000
1725099139,3cyclictest5948-21head11:20:203
1725099139,3cyclictest15432-21cut11:45:133
1725099139,2cyclictest9941-21grep12:20:133
1725099139,2cyclictest5690-21grep09:00:143
1725099132,10cyclictest13541-21users11:05:263
1725099131,3cyclictest475-21in:imuxsock07:20:003
17250991311,1cyclictest0-21swapper/311:14:533
17250991310,2cyclictest9008-21df10:25:123
17250991310,2cyclictest29710-21grep10:55:263
17250991310,2cyclictest29271-21sed11:15:163
17250991310,2cyclictest28958-21head07:30:253
17250991310,2cyclictest28512-21wc12:30:203
17250991310,2cyclictest27221-21basename09:20:003
17250991310,2cyclictest25242-21sed08:30:253
17250991310,2cyclictest24465-21grep11:50:163
17250991310,2cyclictest24249-21grep09:55:253
17250991310,2cyclictest17225-21tr10:30:143
1725099130,1cyclictest0-21swapper/309:53:103
1724699139,3cyclictest30086-21wc11:15:262
1724699139,3cyclictest22348-21cut07:20:152
1724699139,2cyclictest23488-21sed11:30:232
1724699138,2cyclictest0-21swapper/211:14:102
1724699131,12cyclictest0-21swapper/210:40:122
17246991310,2cyclictest8068-21uname07:55:202
17246991310,2cyclictest30372-21grep07:35:142
17246991310,2cyclictest2887-21cron08:55:012
17246991310,2cyclictest19736-21ntpq12:25:212
17246991310,2cyclictest15214-21cut11:45:122
17246991310,2cyclictest11924-21grep11:05:122
17246991310,2cyclictest10946-21dump-pmu-power11:05:002
1724699130,12cyclictest22544-21grep09:55:132
1724399138,5cyclictest0-21swapper/112:00:291
1724399137,6cyclictest0-21swapper/108:20:141
1724399132,10cyclictest24946-21cat12:10:011
1724399131,10cyclictest0-21swapper/107:10:131
17243991310,3cyclictest0-21swapper/108:35:231
17243991310,2cyclictest28882-21grep07:30:251
17243991310,2cyclictest22090-21cat07:20:001
17243991310,2cyclictest10025-21cron09:10:001
17243991310,2cyclictest0-21swapper/107:24:591
1724099139,3cyclictest9534-21wc12:20:010
1724099139,3cyclictest28164-21cron08:40:000
1724099139,3cyclictest23699-21ls07:20:260
1724099139,3cyclictest18615-21perf08:20:000
1724099139,3cyclictest18311-21tr12:05:210
17240991312,1cyclictest10051-21diskmemload10:41:010
17240991310,2cyclictest6650-21taskset11:40:120
17240991310,2cyclictest30326-21cron10:00:000
17240991310,2cyclictest2994-21head07:45:170
17240991310,2cyclictest20417-21grep12:25:270
17240991310,2cyclictest16550-21wc10:30:010
1724099130,7cyclictest29408-21idleruntime-cro07:35:000
1725099129,2cyclictest7167-21tr10:05:133
1725099129,2cyclictest4043-21cut09:25:133
1725099129,2cyclictest39-21ksoftirqd/308:45:273
1725099129,2cyclictest29128-21cut08:40:163
1725099129,2cyclictest22854-21sort07:20:173
1725099129,2cyclictest17681-21grep08:15:213
1725099129,2cyclictest12575-21sh09:49:223
1725099128,3cyclictest2572-21tail10:40:163
1725099128,2cyclictest9107-21sendmail-msp08:00:003
1725099128,2cyclictest22362-21grep08:25:193
1725099120,1cyclictest0-21swapper/307:50:113
1724699129,2cyclictest8923-21chrt09:46:352
1724699129,2cyclictest6154-21sort09:00:162
1724699129,2cyclictest27032-21sleep07:30:002
1724699129,2cyclictest23910-21tr08:30:142
1724699129,2cyclictest17255-21grep08:15:152
1724699129,2cyclictest16358-21cat08:15:002
1724699129,2cyclictest15175-21head11:25:262
1724699128,3cyclictest31104-21ssh09:40:242
1724699127,2cyclictest225-21systemd-journal08:22:232
1724699122,4cyclictest25286-21wget12:10:132
1724699120,10cyclictest498-21turbostat.cron08:50:002
1724399129,2cyclictest5347-21grep07:50:161
17243991210,1cyclictest0-21swapper/109:00:121
1724099129,2cyclictest9981-21cut12:20:130
1724099129,2cyclictest6075-21ls11:20:210
1724099129,2cyclictest32052-21ssh10:00:190
1724099129,2cyclictest31345-21sed09:40:260
1724099129,2cyclictest29136-21cut08:40:160
1724099129,2cyclictest25885-21cat08:35:010
1724099129,2cyclictest20561-21grep07:15:160
1724099129,2cyclictest11879-21grep10:45:210
1724099128,3cyclictest19986-21head10:50:160
17240991210,2cyclictest13-21ksoftirqd/011:13:280
17240991210,2cyclictest0-21swapper/012:39:550
1724099120,4cyclictest0-21swapper/011:06:480
1724099120,11cyclictest3952-21sh11:20:000
1724099120,11cyclictest27027-21perf07:30:000
1725099119,1cyclictest16814-21grep12:05:133
1725099118,2cyclictest32351-21ssh09:41:063
1725099118,2cyclictest27606-21ssh09:20:073
1725099118,2cyclictest20549-21ssh10:12:493
1725099118,2cyclictest20532-21ssh09:35:063
1725099118,2cyclictest19483-21cut08:20:173
1725099118,2cyclictest17762-21tr07:10:143
1725099118,2cyclictest1633-21taskset11:36:453
1725099118,2cyclictest15016-21taskset09:31:033
1725099118,2cyclictest1096-21sh12:15:023
1725099118,2cyclictest10463-21ssh10:45:113
1725099117,2cyclictest8162-21grep09:05:133
1725099111,1cyclictest0-21swapper/311:01:453
1725099110,8cyclictest0-21swapper/311:55:503
1724699118,2cyclictest9311-21chrt08:00:052
1724699118,2cyclictest8240-21ssh12:00:112
1724699118,2cyclictest7702-21grep10:05:152
1724699118,2cyclictest5809-21grep07:50:212
1724699118,2cyclictest5305-21basename09:00:012
1724699118,2cyclictest4607-21ssh12:16:222
1724699118,2cyclictest31147-21tr10:00:142
1724699118,2cyclictest28845-21ssh11:52:502
1724699118,2cyclictest28575-21sed10:55:172
1724699118,2cyclictest27925-21tr09:20:152
1724699118,2cyclictest26324-21expr08:35:152
1724699118,2cyclictest24836-21ssh09:37:042
1724699118,2cyclictest23998-21grep10:15:132
1724699118,2cyclictest23998-21grep10:15:122
1724699118,2cyclictest22183-21sh09:16:102
1724699118,2cyclictest20142-21tr07:15:142
1724699118,2cyclictest1957-21ssh11:55:342
1724699118,2cyclictest18427-21ssh10:11:102
1724699118,2cyclictest17945-21ssh09:14:332
1724699118,2cyclictest1749-21sleep07:45:002
1724699118,2cyclictest17475-21ssh09:51:362
1724699118,2cyclictest17439-21chrt07:10:082
1724699118,2cyclictest14032-21ssh10:46:362
1724699118,2cyclictest13546-21ssh12:21:412
1724699116,5cyclictest26191-21apache210:20:352
1724399118,2cyclictest7706-21grep07:55:171
1724399114,6cyclictest0-21swapper/108:05:161
1724399114,4cyclictest3120-21sshd08:55:051
1724099118,2cyclictest9349-21sed12:00:190
1724099118,2cyclictest839-21grep10:20:170
1724099118,2cyclictest6703-21sh07:55:000
1724099118,2cyclictest3826-21ssh09:25:120
1724099118,2cyclictest30873-21tr08:45:150
1724099118,2cyclictest28901-21ssh12:11:390
1724099118,2cyclictest28845-21cat09:20:190
1724099118,2cyclictest28810-21cat10:55:180
1724099118,2cyclictest26075-21sh11:32:130
1724099118,2cyclictest24809-21ssh09:18:120
1724099118,2cyclictest23665-21ssh10:15:020
1724099118,2cyclictest23665-21ssh10:15:020
1724099118,2cyclictest21527-21tr08:25:140
1724099118,2cyclictest21080-21ssh11:48:250
1724099118,2cyclictest21032-21taskset10:13:110
1724099118,2cyclictest18231-21sed07:10:160
1724099118,2cyclictest17395-21sh09:14:090
1724099118,2cyclictest14525-21expr08:10:140
1724099118,2cyclictest14397-21expr09:50:150
1724099118,2cyclictest12052-21ssh10:07:490
1724099118,2cyclictest10022-21ssh11:04:090
1724099113,4cyclictest225-21systemd-journal11:30:000
1724099111,8cyclictest27006-21ssh12:30:110
1724099110,10cyclictest7966-21sh09:45:460
1725099108,1cyclictest6475-21ssh11:40:063
1725099108,1cyclictest4165-21taskset07:48:113
1725099108,1cyclictest26683-21taskset07:25:583
1725099108,1cyclictest1415-21taskset07:41:043
1725099107,2cyclictest4761-21sort12:35:253
1725099107,2cyclictest2618-21chrt08:51:353
1725099107,2cyclictest12356-21sort08:05:163
1725099100,9cyclictest25212-21sh10:35:043
1724699108,1cyclictest29557-21sed08:40:192
1724699107,2cyclictest9061-21tr09:05:192
1724699107,2cyclictest28060-21cut07:30:162
1724699106,3cyclictest22011-21grep08:25:172
1724699106,3cyclictest11827-21grep08:05:132
1724399107,2cyclictest32762-21chrt08:47:521
1724399104,4cyclictest28701-21cpuspeed_turbos08:40:141
1724399104,3cyclictest225-21systemd-journal08:10:001
1724399103,6cyclictest0-21swapper/107:40:241
1724399103,4cyclictest225-21systemd-journal08:52:591
1724399101,8cyclictest21835-21diskstats08:25:151
1724099108,1cyclictest9722-21expr08:00:150
1724099108,1cyclictest7356-21expr07:55:140
1724099108,1cyclictest20511-21unixbench_multi08:20:260
1724099107,2cyclictest8058-21cut09:05:130
1724099107,2cyclictest6350-21smartctl09:00:170
1724099107,2cyclictest12580-21cat08:05:170
172509997,1cyclictest9444-21cut08:00:143
172509996,2cyclictest29645-21chrt07:35:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional