You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-14 - 08:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Apr 14, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2786729466,9sleep30-21swapper/319:06:213
2801528876,8sleep00-21swapper/019:08:150
2810928270,8sleep20-21swapper/219:09:292
2800128265,13sleep10-21swapper/119:08:031
30252500,0sleep30-21swapper/321:18:113
28242992824,3cyclictest25-21ksoftirqd/121:52:351
2824299260,25cyclictest0-21swapper/123:35:031
2824599231,2cyclictest32-21ksoftirqd/223:20:122
2824599230,1cyclictest0-21swapper/200:20:312
2824299230,23cyclictest0-21swapper/122:56:231
28249992222,0cyclictest0-21swapper/323:00:473
28249992217,5cyclictest0-21swapper/321:33:093
28245992222,0cyclictest0-21swapper/221:24:572
28245992219,3cyclictest32-21ksoftirqd/200:10:412
2824599220,3cyclictest0-21swapper/222:13:472
2824299221,11cyclictest0-21swapper/123:55:111
2824299220,1cyclictest0-21swapper/121:10:141
2824599211,18cyclictest0-21swapper/219:45:132
2824599211,13cyclictest311rcuc/223:50:462
2824999200,20cyclictest1685-21kworker/3:119:10:073
2824999200,1cyclictest0-21swapper/321:10:133
2824999200,19cyclictest1685-21kworker/3:119:20:583
2824599209,11cyclictest32-21ksoftirqd/200:30:502
2824599209,11cyclictest32-21ksoftirqd/200:30:492
28245992018,2cyclictest32-21ksoftirqd/223:46:592
28245992018,2cyclictest32-21ksoftirqd/221:30:012
28245992017,3cyclictest32-21ksoftirqd/223:04:292
28245992014,6cyclictest32-21ksoftirqd/222:49:252
28245992014,6cyclictest32-21ksoftirqd/222:49:242
28245992011,9cyclictest32-21ksoftirqd/222:37:512
28245992010,10cyclictest32-21ksoftirqd/200:07:012
28242992019,1cyclictest0-21swapper/100:30:151
28242992019,1cyclictest0-21swapper/100:30:141
2824299200,19cyclictest0-21swapper/122:48:341
2824299200,19cyclictest0-21swapper/122:48:341
2823999206,7cyclictest15175-21munin-run23:59:590
2823999204,9cyclictest10099-21sendmail19:40:010
213872200,1sleep20-21swapper/223:44:172
2824999199,9cyclictest2620-21kworker/u8:223:24:533
2824999190,19cyclictest1685-21kworker/3:119:16:403
2824599196,12cyclictest0-21swapper/222:20:132
28245991918,1cyclictest21037-21diskmemload21:34:012
28242991918,1cyclictest0-21swapper/119:55:261
28242991916,2cyclictest0-21swapper/123:13:301
2824299190,3cyclictest0-21swapper/123:48:001
2824299190,18cyclictest0-21swapper/120:10:241
2823999190,1cyclictest0-21swapper/021:25:140
311452182,1sleep221037-21diskmemload23:10:392
2824599185,6cyclictest32-21ksoftirqd/221:49:542
2824599182,6cyclictest32-21ksoftirqd/221:36:112
2824599182,16cyclictest32-21ksoftirqd/200:29:362
2824599181,9cyclictest0-21swapper/220:50:142
28245991813,3cyclictest32-21ksoftirqd/222:50:132
2823999183,14cyclictest0-21swapper/019:55:140
2823999180,2cyclictest0-21swapper/021:38:350
2824999170,13cyclictest0-21swapper/323:45:553
2824599172,11cyclictest0-21swapper/222:07:362
28245991715,2cyclictest32-21ksoftirqd/219:10:162
28245991715,1cyclictest0-21swapper/223:15:262
28245991714,1cyclictest21037-21diskmemload21:20:012
28245991711,6cyclictest32-21ksoftirqd/223:05:392
2824599171,14cyclictest311rcuc/222:43:272
2824299171,8cyclictest0-21swapper/121:15:141
28242991717,0cyclictest0-21swapper/122:37:321
2824299170,3cyclictest0-21swapper/121:21:381
2824299170,17cyclictest0-21swapper/121:47:081
2824299170,14cyclictest0-21swapper/121:34:091
2823999177,9cyclictest27151-21munin-run22:50:010
2823999177,9cyclictest27151-21munin-run22:50:000
2823999170,14cyclictest0-21swapper/023:15:200
2824999160,1cyclictest0-21swapper/323:10:453
2824999160,1cyclictest0-21swapper/322:27:293
28245991615,1cyclictest311rcuc/222:31:572
28245991611,5cyclictest10550irq/122-ahci[0022:19:592
28242991611,4cyclictest10450-21apache222:43:041
2824299160,15cyclictest0-21swapper/123:04:241
2823999160,11cyclictest17688-21sensors00:00:330
2823999160,0cyclictest0-21swapper/000:31:010
2823999160,0cyclictest0-21swapper/000:31:010
2824999150,2cyclictest0-21swapper/300:39:443
2824999150,1cyclictest0-21swapper/323:56:083
28245991514,1cyclictest311rcuc/221:55:242
28245991510,4cyclictest0-21swapper/223:34:012
28242991514,1cyclictest38150irq/131-snd_hda00:20:301
28242991513,1cyclictest0-21swapper/122:15:011
28242991510,3cyclictest16182-21wc23:40:261
2824299150,15cyclictest0-21swapper/123:22:421
28239991511,4cyclictest3570-21apache200:23:490
2823999150,2cyclictest0-21swapper/022:42:150
28249991411,2cyclictest32355-21perf00:09:593
2824599147,7cyclictest32-21ksoftirqd/221:50:132
28245991410,3cyclictest26289-21cat20:10:242
28242991410,3cyclictest32443-21cut00:10:011
28242991410,3cyclictest2930-21cron19:25:011
2824299140,1cyclictest0-21swapper/123:54:571
2824299140,1cyclictest0-21swapper/100:02:111
2824299140,13cyclictest5530-21cat20:35:191
28239991410,3cyclictest14444-21grep19:45:260
2823999140,1cyclictest0-21swapper/021:48:070
2823999140,13cyclictest0-21swapper/022:08:360
131802140,3sleep339-21ksoftirqd/322:21:353
2824999138,4cyclictest28225-21/usr/sbin/munin22:30:273
28249991312,1cyclictest39-21ksoftirqd/319:35:073
28249991311,1cyclictest12183-21ntp_states22:40:193
28249991310,2cyclictest28283-21sed22:50:163
28249991310,2cyclictest18638-21tr00:20:203
28249991310,2cyclictest15748-21grep20:55:253
28249991310,2cyclictest12150-21sh21:23:413
2824599139,3cyclictest18011-21tr19:55:152
2824599138,5cyclictest32-21ksoftirqd/223:27:252
2824599137,6cyclictest311rcuc/222:26:252
2824599131,7cyclictest21533-21/usr/sbin/munin21:10:112
28245991310,2cyclictest8905-21cat19:35:192
28245991310,2cyclictest8552-21ssh21:40:232
28245991310,2cyclictest32-21ksoftirqd/200:20:002
28245991310,2cyclictest23722-21grep20:05:222
2824599130,2cyclictest0-21swapper/200:04:502
2824599130,12cyclictest9151-21grep22:00:182
2824599130,10cyclictest7180-21missed_timers23:35:202
2824599130,0cyclictest0-21swapper/223:55:012
2824299131,2cyclictest0-21swapper/122:00:111
28242991311,2cyclictest0-21swapper/123:29:451
28242991310,2cyclictest531-21ls19:20:011
28242991310,2cyclictest27694-21tr20:15:161
28242991310,2cyclictest1586-21ssh21:55:261
2824299130,12cyclictest0-21swapper/120:40:221
2823999139,3cyclictest519-21grep20:25:170
2823999139,3cyclictest20233-21tr20:00:140
2823999135,7cyclictest28384-21munin-run23:10:010
2823999132,10cyclictest30462-21sensors23:10:220
2823999131,9cyclictest25528-21/usr/sbin/munin00:05:200
2823999131,6cyclictest5486-21awk23:35:000
28239991310,2cyclictest26611-21grep00:25:170
28239991310,2cyclictest25703-21wc21:50:260
28239991310,2cyclictest24462-21sort21:30:270
28239991310,2cyclictest22533-21sed23:25:170
28239991310,2cyclictest19986-21grep22:25:260
2823999130,2cyclictest0-21swapper/023:46:500
2824999129,2cyclictest9631-21grep20:45:153
2824999129,2cyclictest534-21grep21:35:263
2824999129,2cyclictest3321-21tr22:35:203
2824999129,2cyclictest30467-21grep20:20:183
2824999129,2cyclictest25857-21tr00:25:143
2824999129,2cyclictest22773-21grep20:05:153
2824999129,2cyclictest21037-21diskmemload22:12:223
2824999129,2cyclictest17171-21ssh23:40:583
2824999129,2cyclictest17016-21tail19:50:263
2824999129,2cyclictest16888-21sendmail-msp00:19:593
2824999129,2cyclictest16693-21grep00:00:173
2824999129,2cyclictest15160-21grep21:45:003
2824999129,2cyclictest13538-21cut19:45:163
2824999128,3cyclictest3454-21wc20:30:243
2824999128,3cyclictest2230-21ssh21:55:423
28249991210,1cyclictest5709-21grep23:15:183
2824599129,2cyclictest8628-21sed20:40:272
2824599129,2cyclictest18797-21cat21:05:012
2824599129,2cyclictest18797-21cat21:05:002
2824599129,2cyclictest10504-21strings20:45:202
2824599128,3cyclictest0-21swapper/222:56:442
2824599124,7cyclictest3847-21fschecks_time19:25:152
28245991211,1cyclictest32-21ksoftirqd/200:39:082
2824299129,2cyclictest6745-21sed23:15:271
2824299129,2cyclictest30325-21tr23:30:141
2824299129,2cyclictest20480-21ssh23:05:131
2824299129,2cyclictest20025-21cut21:05:201
2824299129,2cyclictest1670-21head00:10:201
2824299129,2cyclictest16054-21cut19:50:171
2824299129,2cyclictest10602-21grep00:35:161
2824299129,2cyclictest10055-21grep20:45:181
2824299128,3cyclictest9830-21cut00:15:171
2824299128,3cyclictest18129-21tr22:05:211
2824299127,4cyclictest487-21avahi-daemon00:25:031
2824299120,9cyclictest0-21swapper/121:38:501
2824299120,7cyclictest0-21swapper/120:05:131
2823999129,2cyclictest7725-21grep20:40:170
2823999129,2cyclictest5429-21wc22:55:250
2823999129,2cyclictest29587-21munin-plugin-st20:19:590
2823999129,2cyclictest28722-21grep19:10:160
2823999129,2cyclictest24611-21turbostat.cron20:10:000
2823999129,2cyclictest1372-21grep19:20:140
2823999129,2cyclictest11842-21grep20:50:140
2823999129,2cyclictest11145-21head00:35:180
2823999129,2cyclictest11009-21grep20:45:270
2823999129,1cyclictest0-21swapper/021:24:220
2823999128,3cyclictest524-21cut00:10:130
2823999128,3cyclictest31832-21grep21:15:280
2823999128,2cyclictest17796-21wc21:00:220
2823999128,2cyclictest17796-21wc21:00:210
2823999120,1cyclictest0-21swapper/023:24:310
2824999119,1cyclictest11807-21grep20:50:143
2824999119,1cyclictest0-21swapper/322:06:073
2824999118,2cyclictest9122-21ssh23:36:183
2824999118,2cyclictest7545-21ssh00:14:193
2824999118,2cyclictest4831-21tr20:35:163
2824999118,2cyclictest3509-21ssh22:15:463
2824999118,2cyclictest31860-21cat23:30:253
2824999118,2cyclictest25948-21ssh23:08:083
2824999118,2cyclictest23223-21ssh21:49:513
2824999118,2cyclictest2322-21expr00:30:163
2824999118,2cyclictest2322-21expr00:30:153
2824999118,2cyclictest21750-21wget23:25:143
2824999118,2cyclictest19621-21grep21:05:173
2824999118,2cyclictest17848-21basename19:55:143
2824999118,2cyclictest17497-21cat21:00:193
2824999118,2cyclictest17497-21cat21:00:183
2824999118,2cyclictest16295-21sed21:25:273
2824999118,2cyclictest13813-21ssh22:03:073
2824999118,2cyclictest1206-21ssh23:50:463
2824999111,6cyclictest19233-21/usr/sbin/munin22:45:133
2824999111,6cyclictest19233-21/usr/sbin/munin22:45:133
2824599119,2cyclictest32-21ksoftirqd/220:30:092
2824599119,2cyclictest0-21swapper/219:30:152
2824599118,2cyclictest4848-21grep20:35:152
2824599118,2cyclictest323-21sed20:25:162
2824599118,2cyclictest30967-21basename19:15:152
2824599118,2cyclictest20394-21cat21:05:252
2824599118,2cyclictest16448-21grep19:50:202
2824299119,1cyclictest28692-21grep19:10:151
2824299118,2cyclictest8924-21cat19:35:201
2824299118,2cyclictest29350-21ssh22:12:101
2824299118,2cyclictest29068-21ssh22:30:521
2824299118,2cyclictest18979-21sh22:25:171
2824299118,2cyclictest17940-21ssh21:26:411
2824299118,2cyclictest16305-21wc21:00:011
2824299118,2cyclictest10952-21ssh21:41:501
2824299116,3cyclictest27663-21apt-get22:50:141
2823999118,2cyclictest7717-21grep21:40:160
2823999118,2cyclictest7390-21grep23:35:220
2823999118,2cyclictest3495-21ssh23:52:330
2823999118,2cyclictest3349-21head22:35:200
2823999118,2cyclictest32353-21chrt22:52:330
2823999118,2cyclictest31091-21tr19:15:150
2823999118,2cyclictest28112-21ssh22:11:110
2823999118,2cyclictest27900-21ssh21:13:500
2823999118,2cyclictest27851-21open_inodes22:30:220
2823999118,2cyclictest2290-21ssh22:15:210
2823999118,2cyclictest21140-21wc21:10:000
2823999118,2cyclictest16275-21basename21:00:000
2823999118,2cyclictest15211-21sort23:40:170
2823999118,2cyclictest12947-21sed23:00:170
2823999113,6cyclictest32304-21ssh21:55:140
2823999112,8cyclictest16884-21awk00:20:000
2823999110,1cyclictest0-21swapper/022:02:130
2824999108,1cyclictest7353-21ls20:40:153
2824999108,1cyclictest73422sleep319:30:493
2824999108,1cyclictest4774-21cat19:25:253
2824999108,1cyclictest3546-21cat22:55:103
2824999108,1cyclictest32240-21apache_accesses20:25:123
2824999108,1cyclictest27541-21awk20:15:153
2824999108,1cyclictest25337-21expr20:10:123
2824999108,1cyclictest25195-21sh21:50:213
2824999108,1cyclictest22359-21cat20:04:593
2824999107,2cyclictest11908-21cat19:40:263
2824599108,1cyclictest30985-21awk20:20:212
2824599108,1cyclictest14271-21cut20:55:142
2824599107,2cyclictest27814-21expr20:15:162
2824599107,2cyclictest21354-21tr20:00:212
2824599107,2cyclictest1156-21expr19:20:132
2824599105,2cyclictest225-21systemd-journal19:45:002
2824299108,1cyclictest9590-21ssh22:20:111
2824299108,1cyclictest3965-21taskset20:30:471
2824299108,1cyclictest18639-21chrt21:04:591
2824299108,1cyclictest18639-21chrt21:04:581
2824299108,1cyclictest145762sleep119:46:141
2824299108,1cyclictest13536-21chrt20:50:451
2824299107,2cyclictest5821-21tr19:30:161
2824299107,2cyclictest5262-21taskset19:29:341
2824299107,2cyclictest31918-21chrt20:24:241
2824299107,2cyclictest10690-21tr19:40:151
2824299106,3cyclictest843-21wc20:25:191
2824299100,9cyclictest21214-21needreboot20:00:191
2823999108,1cyclictest7124-21sed19:30:270
2823999108,1cyclictest4661-21sed19:25:240
2823999108,1cyclictest12305-21chrt19:42:320
2823999107,2cyclictest5169-21fschecks_count20:35:180
2823999107,2cyclictest4278-21taskset20:34:490
2823999107,2cyclictest30927-21wc20:20:210
2823999107,2cyclictest25139-21tr20:10:130
2823999107,2cyclictest16933-21sort19:50:250
2823999107,2cyclictest10536-21tail22:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional