You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 29, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2690729165,8sleep00-21swapper/007:09:520
2668028860,9sleep30-21swapper/307:06:573
2671128661,9sleep20-21swapper/207:07:212
2552427665,6sleep10-21swapper/107:05:191
2700599280,27cyclictest0-21swapper/110:05:141
2700899260,26cyclictest0-21swapper/212:05:572
27005992525,0cyclictest0-21swapper/110:50:571
2700899230,23cyclictest0-21swapper/210:24:102
2700299222,17cyclictest8700-21kworker/u8:110:45:120
2700599210,20cyclictest0-21swapper/107:25:141
2700599210,1cyclictest0-21swapper/112:37:221
108682210,0sleep2311rcuc/210:40:252
40512200,0sleep30-21swapper/310:55:373
27012991919,0cyclictest0-21swapper/310:13:393
27012991914,4cyclictest0-21swapper/309:50:013
2700899190,17cyclictest0-21swapper/211:45:412
2701299180,17cyclictest0-21swapper/309:31:303
27008991816,1cyclictest0-21swapper/209:50:132
2700899180,18cyclictest0-21swapper/209:42:122
2700899180,17cyclictest0-21swapper/211:14:562
27005991816,1cyclictest20011-21diskmemload10:04:291
2700599181,1cyclictest16020-21ssh10:44:241
2700599180,18cyclictest0-21swapper/111:47:081
2700599180,0cyclictest0-21swapper/109:26:261
2700599180,0cyclictest0-21swapper/109:26:251
2700299180,2cyclictest0-21swapper/011:14:340
2700899170,17cyclictest0-21swapper/212:29:042
2700599177,3cyclictest225-21systemd-journal07:43:121
2700599170,2cyclictest141rcu_preempt11:04:511
2700299170,3cyclictest0-21swapper/009:20:050
2700899164,10cyclictest1298-21ls10:55:012
2700899160,16cyclictest0-21swapper/209:49:342
2700899160,15cyclictest0-21swapper/211:38:532
27005991612,3cyclictest13814-21ssh09:45:001
2700599161,1cyclictest17110-21ssh10:45:051
2700599160,1cyclictest0-21swapper/109:36:031
2700599160,16cyclictest0-21swapper/112:19:531
27002991615,1cyclictest0-21swapper/010:37:160
27012991510,4cyclictest55850irq/129-enp2s0-09:17:463
27008991513,1cyclictest0-21swapper/207:50:212
2700899150,15cyclictest0-21swapper/209:29:572
2700899150,15cyclictest0-21swapper/209:29:572
2700899150,13cyclictest0-21swapper/211:53:252
2700899150,0cyclictest0-21swapper/212:19:162
27005991514,0cyclictest0-21swapper/110:34:561
27005991513,1cyclictest0-21swapper/111:15:161
27005991511,2cyclictest20011-21diskmemload10:12:271
2700599150,0cyclictest0-21swapper/111:21:071
2700299157,4cyclictest225-21systemd-journal09:03:290
27002991511,3cyclictest4262-21cat07:30:010
2700299151,13cyclictest22688-21ssh09:10:330
2700299151,12cyclictest778-21/usr/sbin/munin10:15:210
2700299150,14cyclictest24115-21chrt11:08:080
2700299150,14cyclictest0-21swapper/011:34:510
27012991411,2cyclictest4763-21sort08:35:263
27008991411,2cyclictest18438-21sendmail-msp08:00:002
27008991411,2cyclictest13362-21grep11:40:162
27008991410,2cyclictest15543-21ssh10:05:162
2700599142,12cyclictest20011-21diskmemload11:32:171
27005991414,0cyclictest0-21swapper/112:29:111
27005991413,1cyclictest0-21swapper/112:05:181
27005991413,1cyclictest0-21swapper/109:33:401
2700599140,14cyclictest0-21swapper/111:54:021
2700599140,14cyclictest0-21swapper/111:05:121
27002991413,1cyclictest0-21swapper/012:02:220
27002991410,3cyclictest4899-21head11:15:270
27002991410,2cyclictest10834-21grep08:50:160
2700299140,14cyclictest0-21swapper/011:43:520
2701299139,3cyclictest6003-21grep08:40:163
2701299139,3cyclictest28137-21sh08:19:593
2701299139,3cyclictest2393-21cron11:15:003
2701299133,8cyclictest20011-21diskmemload12:25:533
2701299131,8cyclictest0-21swapper/307:50:153
27012991310,2cyclictest672-21cat10:15:203
27012991310,2cyclictest6160-21grep11:35:283
27012991310,2cyclictest29682-21tr11:50:143
27012991310,2cyclictest2591-21grep08:30:263
27012991310,2cyclictest24763-21grep10:30:143
27012991310,2cyclictest20259-21grep11:25:153
27012991310,2cyclictest16245-21sed09:45:273
27012991310,2cyclictest15682-21cron12:40:013
27012991310,2cyclictest10770-21ssh11:00:173
2700899135,4cyclictest225-21systemd-journal08:24:592
27008991310,2cyclictest8815-21cron07:40:012
27008991310,2cyclictest425-21sed09:55:272
27008991310,2cyclictest31402-21head12:10:192
27008991310,2cyclictest14315-21df12:00:162
27008991310,2cyclictest1393-21grep10:15:262
27008991310,2cyclictest11217-21sh07:45:002
2700899130,13cyclictest0-21swapper/210:57:592
2700899130,12cyclictest0-21swapper/212:36:402
2700599139,3cyclictest20789-21ssh09:10:171
2700599139,2cyclictest12793-21cron08:55:001
2700599138,3cyclictest23331-21wc08:09:591
27005991310,2cyclictest8918-21wc10:20:181
27005991310,2cyclictest27786-21grep07:10:151
27005991310,2cyclictest1460-21grep10:15:281
2700599130,13cyclictest0-21swapper/111:26:071
2700599130,11cyclictest0-21swapper/110:58:041
2700299139,3cyclictest7117-21cut08:40:260
2700299139,3cyclictest7068-21latency_hist12:35:000
2700299139,3cyclictest27944-21grep07:10:160
27002991310,3cyclictest0-21swapper/010:23:220
27002991310,2cyclictest9136-21cut12:35:220
27002991310,2cyclictest6457-21cat10:00:010
27002991310,2cyclictest32719-21head08:25:270
27002991310,2cyclictest1772-21wc07:25:000
27002991310,2cyclictest13695-21latency_hist09:45:000
2700299130,1cyclictest141rcu_preempt12:05:390
2700299130,12cyclictest0-21swapper/009:35:080
288072120,0chrt0-21swapper/012:28:110
2701299129,2cyclictest5985-21sh09:40:143
2701299129,2cyclictest5707-21head09:20:183
2701299129,2cyclictest31950-21tr12:30:163
2701299129,2cyclictest30610-21grep08:24:593
2701299129,2cyclictest28413-21chrt11:49:393
2701299129,2cyclictest26134-21grep10:50:143
2701299129,2cyclictest21243-21grep09:10:193
2701299129,2cyclictest2040-21cut07:25:153
2701299129,2cyclictest19583-21grep09:05:273
2701299129,2cyclictest19396-21cat11:25:013
2701299129,2cyclictest16792-21df10:25:153
2701299129,2cyclictest1595-21ssh11:33:203
2701299129,2cyclictest15292-21grep12:00:253
2701299129,2cyclictest13104-21cut08:55:123
2701299129,2cyclictest0-21swapper/310:01:013
2701299129,1cyclictest26993-21cyclictest10:38:493
2701299128,3cyclictest11570-21ssh10:40:583
2701299122,8cyclictest3656-21fschecks_time11:15:163
2700899129,2cyclictest6228-21tail11:55:172
2700899129,2cyclictest5569-21grep07:30:222
2700899129,2cyclictest30708-21tr07:15:212
2700899129,2cyclictest28945-21grep07:10:262
2700899129,2cyclictest24436-21cut10:10:162
2700899129,2cyclictest23570-21grep09:30:282
2700899129,2cyclictest16648-21cat12:20:252
2700899128,3cyclictest26752-21head08:15:172
2700899128,3cyclictest20800-21sed11:25:192
2700899125,4cyclictest11654-21latency08:50:192
2700899123,7cyclictest0-21swapper/211:25:002
2700599129,2cyclictest31805-21head08:25:181
2700599129,2cyclictest31719-21head09:55:171
2700599129,2cyclictest14676-21grep12:00:191
2700599120,12cyclictest0-21swapper/112:10:211
2700599120,0cyclictest0-21swapper/111:38:021
2700299129,2cyclictest9380-21wc08:45:190
2700299129,2cyclictest6251-21sh11:55:170
2700299129,2cyclictest31969-21ls12:10:230
2700299129,2cyclictest27916-21ssh10:12:160
2700299129,2cyclictest24499-21grep09:50:260
2700299129,2cyclictest24376-21grep08:10:160
2700299129,2cyclictest13781-21tail08:55:150
2700299129,2cyclictest12577-21cat07:45:190
2700299128,3cyclictest29162-21head08:20:160
2700299128,3cyclictest24630-21ssh10:30:120
2700299124,4cyclictest28959-21ssh09:15:130
27002991210,1cyclictest4952-21grep08:35:250
27002991210,1cyclictest14011-21grep09:25:170
27002991210,1cyclictest14011-21grep09:25:160
2700299120,2cyclictest0-21swapper/011:48:090
2700299120,1cyclictest0-21swapper/011:50:110
2700299120,11cyclictest30713-21grep07:15:210
2700299120,11cyclictest14670-21cron10:04:590
2701299118,2cyclictest9049-21head08:45:173
2701299118,2cyclictest8580-21ssh12:15:293
2701299118,2cyclictest7335-21cut07:35:173
2701299118,2cyclictest5105-21ssh11:55:033
2701299118,2cyclictest463-21ssh12:10:433
2701299118,2cyclictest32637-21grep08:25:273
2701299118,2cyclictest32151-21wget07:20:143
2701299118,2cyclictest28271-21sort07:10:193
2701299118,2cyclictest23920-21cut08:10:153
2701299118,2cyclictest20154-21ssh10:46:073
2701299118,2cyclictest19462-21head08:00:183
2701299118,2cyclictest19387-21ssh12:22:223
2701299118,2cyclictest18628-21ssh11:05:113
2701299118,2cyclictest1854-21ssh09:37:143
2701299118,2cyclictest15347-21ssh10:05:153
2701299118,2cyclictest13337-21sort11:40:163
2701299118,2cyclictest12142-21grep07:45:173
2701299118,2cyclictest11042-21taskset07:42:563
2701299117,3cyclictest30390-21head07:15:183
2701299117,3cyclictest23154-21cron08:09:593
2701299110,6cyclictest0-21swapper/309:25:183
2701299110,6cyclictest0-21swapper/309:25:183
2701299110,10cyclictest24055-21ssh12:05:253
2700899118,2cyclictest8835-21ssh09:21:592
2700899118,2cyclictest877-21cat10:35:152
2700899118,2cyclictest6871-21ssh11:16:572
2700899118,2cyclictest5980-21tr08:40:162
2700899118,2cyclictest31751-21head08:25:182
2700899118,2cyclictest31616-21sh12:30:122
2700899118,2cyclictest30713-21cat09:15:242
2700899118,2cyclictest30328-21grep11:30:282
2700899118,2cyclictest24464-21ssh10:30:072
2700899118,2cyclictest23921-21expr08:10:142
2700899118,2cyclictest2366-21python08:30:252
2700899118,2cyclictest21678-21ssh11:06:142
2700899118,2cyclictest212012sleep209:10:182
2700899118,2cyclictest19306-21grep08:00:182
2700899118,2cyclictest1802-21ssh09:37:122
2700899118,2cyclictest14083-21ssh11:01:542
2700899118,2cyclictest13716-21sort08:55:152
2700899118,2cyclictest10633-21ssh10:01:522
2700899111,9cyclictest11491-21runrttasks07:45:132
2700899110,11cyclictest0-21swapper/210:29:532
2700899110,10cyclictest18608-21sh10:45:202
2700599118,2cyclictest8843-21sleep07:40:001
2700599118,2cyclictest8592-21ssh09:21:461
2700599118,2cyclictest6458-21ls07:35:001
2700599118,2cyclictest6451-21grep08:40:191
2700599118,2cyclictest6192-21grep11:55:171
2700599118,2cyclictest32279-21tr07:20:151
2700599118,2cyclictest28693-21tr08:20:141
2700599118,2cyclictest20011-21mailstats08:00:231
2700599118,2cyclictest20009-21latency_hist09:10:001
2700599118,2cyclictest17572-21ssh10:25:211
2700599118,2cyclictest16312-21ssh09:45:291
2700599118,2cyclictest16012-21grep09:00:161
2700599118,2cyclictest14695-21sed11:40:261
2700599118,2cyclictest12192-21head07:45:171
2700599118,2cyclictest1209-21tr08:30:151
2700599117,3cyclictest32435-21ssh11:13:091
2700599117,3cyclictest30405-21munin-run09:55:001
2700599117,3cyclictest16963-21grep07:55:181
2700599117,3cyclictest16850-21sed12:20:261
2700599117,2cyclictest16028-21perf07:55:011
2700599112,3cyclictest0-21swapper/112:30:151
2700599110,10cyclictest6260-21sh10:38:221
2700599110,10cyclictest0-21swapper/109:15:141
2700299118,2cyclictest6586-21awk12:15:150
2700299118,2cyclictest6374-21chrt07:34:100
2700299118,2cyclictest6207-21sed11:35:270
2700299118,2cyclictest4481-21ssh10:55:560
2700299118,2cyclictest27188-21ssh10:50:210
2700299118,2cyclictest27138-21head08:15:200
2700299118,2cyclictest2616-21tail08:30:270
2700299118,2cyclictest24052-21ssh09:30:490
2700299118,2cyclictest22475-21grep08:05:270
2700299118,2cyclictest18549-21sort09:05:170
2700299118,2cyclictest17835-21ssh10:05:550
2700299118,2cyclictest17090-21ssh12:20:320
2700299118,2cyclictest17047-21tail07:55:180
2700299118,2cyclictest16014-21cron07:55:010
2700299118,2cyclictest15759-21grep09:45:230
2700299118,2cyclictest15148-21ssh10:43:440
2700299118,2cyclictest12422-21ssh11:20:210
2700299118,2cyclictest12169-21sed11:00:280
2700299118,2cyclictest0-21swapper/007:35:110
2700299110,1cyclictest0-21swapper/011:26:450
2700299110,10cyclictest17569-21sh10:25:210
2701299108,1cyclictest30870-21ssh09:55:123
2701299108,1cyclictest176162sleep309:04:583
2701299108,1cyclictest16713-21ls07:55:163
2701299108,1cyclictest11142-21expr08:50:153
2701299108,1cyclictest10111-21chrt10:20:383
2700899108,1cyclictest8798-21expr08:45:142
2700899108,1cyclictest4424-21cat08:35:202
2700899108,1cyclictest3886-21taskset07:27:442
2700899108,1cyclictest22440-21grep08:05:232
2700899108,1cyclictest16115-21forks09:00:172
2700899100,9cyclictest899-21sendmail_mailqu07:20:232
2700899100,9cyclictest18389-21sed09:05:162
2700599108,1cyclictest83212sleep108:45:111
2700599107,2cyclictest3801-21expr08:35:141
2700599107,2cyclictest31308-21grep07:15:281
2700599107,2cyclictest26744-21grep08:15:171
2700599107,2cyclictest23599-21chrt08:10:011
2700599107,2cyclictest14163-21grep08:55:181
2700299108,1cyclictest20651-21seq08:03:190
2700299106,3cyclictest11225-21munin-plugin-st07:45:010
270129997,1cyclictest5427-21needreboot07:30:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional