You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 18:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 16, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2620121210,0sleep00-21swapper/009:50:300
2765829983,12sleep00-21swapper/007:09:120
2740229376,12sleep30-21swapper/307:05:523
2758129179,8sleep20-21swapper/207:08:122
2766928964,15sleep10-21swapper/107:09:201
27902550,0sleep30-21swapper/311:50:383
2780199240,24cyclictest0-21swapper/209:30:282
2779999232,1cyclictest0-21swapper/011:55:130
2780399220,13cyclictest225-21systemd-journal12:29:083
2780399220,0cyclictest0-21swapper/312:09:233
2780399220,0cyclictest0-21swapper/312:09:223
2780399210,12cyclictest3072-21date10:35:003
2780199210,20cyclictest0-21swapper/212:16:402
2780199210,16cyclictest0-21swapper/210:37:302
27800992119,1cyclictest25-21ksoftirqd/110:50:141
2780099210,2cyclictest141rcu_preempt09:20:221
2779999210,1cyclictest0-21swapper/009:16:330
27801992019,1cyclictest0-21swapper/210:59:042
2780199190,19cyclictest0-21swapper/211:13:092
2780199190,18cyclictest0-21swapper/209:43:272
2780099190,18cyclictest0-21swapper/110:12:111
2780399185,2cyclictest141rcu_preempt09:16:003
2780199180,1cyclictest0-21swapper/212:00:552
2780199180,16cyclictest0-21swapper/211:03:252
2780199180,16cyclictest0-21swapper/210:51:222
2780099180,18cyclictest0-21swapper/111:50:011
2780099180,17cyclictest0-21swapper/109:25:241
27799991818,0cyclictest0-21swapper/010:38:240
2780399175,5cyclictest14805-21apache_volume11:20:143
27801991713,3cyclictest0-21swapper/210:45:292
2780199170,3cyclictest0-21swapper/209:20:072
2780199170,17cyclictest0-21swapper/211:23:572
27799991716,1cyclictest13-21ksoftirqd/010:52:310
27803991615,1cyclictest0-21swapper/312:01:353
2780399160,1cyclictest0-21swapper/312:34:153
2780199168,1cyclictest32-21ksoftirqd/211:19:272
2780199161,1cyclictest0-21swapper/212:37:212
2780199160,2cyclictest0-21swapper/209:50:152
2780199160,14cyclictest0-21swapper/211:30:242
2780099160,16cyclictest0-21swapper/112:17:351
2779999160,1cyclictest0-21swapper/010:22:440
2779999160,16cyclictest0-21swapper/011:13:100
2780399158,6cyclictest39-21ksoftirqd/311:02:303
27803991514,1cyclictest0-21swapper/310:48:073
2780399151,14cyclictest0-21swapper/311:27:353
2780199151,0cyclictest141rcu_preempt10:42:172
2780099158,6cyclictest11572-21sed10:20:221
27800991513,1cyclictest0-21swapper/112:36:531
27800991512,2cyclictest241rcuc/109:40:591
2779999150,1cyclictest0-21swapper/010:00:130
2780399147,5cyclictest23157-21grep09:30:173
27803991410,2cyclictest15060-21ssh09:45:003
2780399140,13cyclictest17368-21cut10:05:143
2780199140,3cyclictest0-21swapper/210:15:592
2780199140,14cyclictest0-21swapper/210:29:312
2780099144,7cyclictest28379-21turbostat10:50:001
2780099144,3cyclictest225-21systemd-journal09:18:291
27800991413,1cyclictest0-21swapper/107:25:271
27800991412,1cyclictest0-21swapper/111:38:011
27800991412,1cyclictest0-21swapper/110:37:061
27800991411,2cyclictest20582-21cron09:10:001
27800991410,3cyclictest4908-21cut08:35:191
2780099140,13cyclictest22876-21wc09:10:261
2779999145,1cyclictest16072-21apache_accesses11:40:120
2779999142,7cyclictest22202-21/usr/sbin/munin10:45:260
27799991411,2cyclictest5387-21taskset09:20:070
2780399139,3cyclictest29003-21head07:10:173
2780399139,2cyclictest3677-21grep10:15:253
2780399138,3cyclictest26381-21head08:10:263
2780399137,6cyclictest16850-21apache210:43:233
27803991310,2cyclictest9342-21sed07:35:263
27803991310,2cyclictest9094-21grep11:55:143
27803991310,2cyclictest30979-21tail11:10:153
27803991310,2cyclictest2909-21tr08:30:213
27803991310,2cyclictest2764-21grep12:10:173
27803991310,2cyclictest24391-21sh08:10:003
2780199138,1cyclictest32-21ksoftirqd/209:35:472
2780199131,11cyclictest0-21swapper/208:50:222
2780199131,11cyclictest0-21swapper/208:50:222
27801991310,2cyclictest18026-21grep11:40:272
2780199130,7cyclictest0-21swapper/211:59:022
2780199130,2cyclictest0-21swapper/207:15:122
2780199130,13cyclictest0-21swapper/211:27:292
2780099139,3cyclictest2483-21grep08:30:161
2780099138,4cyclictest50232sleep110:16:241
27800991312,1cyclictest20598-21diskmemload10:40:561
2780099131,11cyclictest6040-21sh09:58:271
2780099131,10cyclictest20598-21diskmemload12:05:071
2780099131,10cyclictest20598-21diskmemload12:05:071
27800991310,2cyclictest9341-21grep07:35:261
27800991310,2cyclictest31605-21cut09:35:171
27800991310,2cyclictest15230-21grep11:00:261
27800991310,2cyclictest12499-21sed08:50:221
27800991310,2cyclictest12499-21sed08:50:221
2780099130,2cyclictest0-21swapper/111:10:461
2780099130,1cyclictest0-21swapper/112:21:281
2780099130,13cyclictest0-21swapper/111:58:561
2779999138,4cyclictest1826-21ntp_states11:50:220
2779999134,7cyclictest25851-21apache_accesses12:05:110
2779999134,7cyclictest25851-21apache_accesses12:05:110
2779999133,5cyclictest1142-21runrttasks11:37:540
27799991312,1cyclictest0-21swapper/012:10:430
27799991310,2cyclictest8060-21run-parts09:59:590
27799991310,2cyclictest29751-21ssh11:28:520
27799991310,2cyclictest14595-21cron07:49:590
2779999130,1cyclictest0-21swapper/009:35:190
2779999130,12cyclictest3470-21grep08:30:260
2780399129,2cyclictest5847-21df07:30:153
2780399129,2cyclictest4633-21df08:35:163
2780399129,2cyclictest29653-21grep10:50:183
2780399129,2cyclictest2487-21chrt09:36:583
2780399129,2cyclictest22147-21ssh09:48:543
2780399129,2cyclictest21353-21tr09:10:133
2780399129,2cyclictest19485-21head10:25:173
2780399129,2cyclictest18211-21cron09:05:003
2780399129,2cyclictest16130-21ssh10:23:293
2780399129,2cyclictest13888-21grep08:55:143
2780399129,2cyclictest12068-21head08:50:173
2780399129,2cyclictest12068-21head08:50:173
2780399128,3cyclictest29148-21cron08:20:013
2780399120,11cyclictest7185-21head11:15:173
2780199129,2cyclictest3914-21cut07:25:202
2780199129,2cyclictest24389-21cron08:10:002
2780199127,4cyclictest537-21snmpd09:55:522
2780199121,9cyclictest30070-21df09:15:152
27801991211,1cyclictest0-21swapper/210:10:062
27801991210,1cyclictest4788-21cut08:35:172
2780199120,1cyclictest0-21swapper/212:09:422
2780199120,1cyclictest0-21swapper/212:09:422
2780199120,1cyclictest0-21swapper/211:52:462
2780199120,1cyclictest0-21swapper/211:35:022
2780199120,11cyclictest7292-21cut08:40:172
2780199120,11cyclictest0-21swapper/209:29:202
2780099129,2cyclictest6066-21head07:30:161
2780099129,2cyclictest32493-21sort08:25:151
2780099129,2cyclictest27537-21grep12:25:161
2780099129,2cyclictest25887-21chrt08:10:211
2780099129,2cyclictest25369-21grep11:45:161
2780099129,2cyclictest19628-21cat08:00:011
2780099129,2cyclictest13637-21chrt10:02:381
2780099129,2cyclictest0-21swapper/112:01:211
2780099128,3cyclictest2753-21head12:10:181
2780099128,3cyclictest17601-21tail09:45:261
2780099125,3cyclictest25-21ksoftirqd/109:51:391
2780099121,11cyclictest0-21swapper/111:21:451
2780099121,10cyclictest0-21swapper/111:06:321
2780099120,1cyclictest0-21swapper/110:34:441
2779999129,2cyclictest8680-21cat09:40:260
2779999129,2cyclictest5255-21grep12:30:250
2779999129,2cyclictest28651-21grep08:15:260
2779999129,2cyclictest25357-21grep11:45:160
2779999129,2cyclictest19443-21cat09:05:190
2779999129,2cyclictest18942-21grep10:25:140
2779999129,2cyclictest1808-21tr10:15:130
2779999129,2cyclictest17373-21grep07:55:140
2779999129,2cyclictest15943-21grep09:25:250
2779999129,2cyclictest15324-21grep08:55:260
2779999129,2cyclictest13964-21cut11:00:150
2779999129,2cyclictest12310-21cat08:50:200
2779999129,2cyclictest12310-21cat08:50:190
2779999127,5cyclictest13-21ksoftirqd/010:10:200
2779999123,3cyclictest31572-21idleruntime-cro08:25:010
2779999121,8cyclictest0-21swapper/009:14:120
27799991211,1cyclictest0-21swapper/010:59:050
2779999120,9cyclictest0-21swapper/010:05:140
2779999120,11cyclictest28193-21ssh09:33:190
2780399119,1cyclictest22522-21taskset12:21:383
2780399118,2cyclictest6008-21ssh10:35:493
2780399118,2cyclictest5415-21ssh09:20:083
2780399118,2cyclictest4842-21ssh10:55:123
2780399118,2cyclictest3391-21ls07:25:163
2780399118,2cyclictest32285-21ssh09:55:043
2780399118,2cyclictest31573-21ssh11:30:083
2780399118,2cyclictest31367-21head07:15:153
2780399118,2cyclictest30227-21ssh09:53:373
2780399118,2cyclictest29476-21grep08:20:133
2780399118,2cyclictest28208-21ssh10:10:473
2780399118,2cyclictest25388-21sed11:45:173
2780399118,2cyclictest24624-21ssh11:06:073
2780399118,2cyclictest21719-21ssh11:43:123
2780399118,2cyclictest20683-21head08:00:173
2780399118,2cyclictest17947-21cut07:55:163
2780399118,2cyclictest14778-21tail09:25:163
2780399118,2cyclictest13293-21ssh10:02:223
2780399118,2cyclictest11812-21apache_volume12:35:143
2780399118,2cyclictest11404-21ssh11:36:473
2780399118,2cyclictest10931-21df12:15:163
2780199119,2cyclictest13975-21wc07:45:242
2780199119,1cyclictest9068-21cut10:00:142
2780199119,1cyclictest3677-21tr12:30:132
2780199119,1cyclictest28239-21head12:25:182
2780199119,1cyclictest18715-21tr09:05:162
2780199119,1cyclictest1805-21grep08:30:142
2780199118,2cyclictest959-21expr07:20:142
2780199118,2cyclictest5427-21cut07:30:132
2780199118,2cyclictest30986-21ssh10:31:552
2780199118,2cyclictest30462-21ssh11:48:292
2780199118,2cyclictest30369-21apt07:15:012
2780199118,2cyclictest30009-21sed08:20:152
2780199118,2cyclictest2782-21grep12:10:182
2780199118,2cyclictest26314-21tail08:10:262
2780199118,2cyclictest23216-21awk11:05:222
2780199118,2cyclictest21755-21taskset08:01:462
2780199118,2cyclictest20484-21grep12:20:222
2780199118,2cyclictest18230-21chrt10:05:182
2780199118,2cyclictest17182-21wc09:45:222
2780199118,2cyclictest11344-21head07:40:222
2780199114,2cyclictest20598-21diskmemload09:10:112
2780199113,2cyclictest10212-21apt-get10:20:132
2780099118,2cyclictest9721-21ssh11:16:311
2780099118,2cyclictest3906-21ssh11:32:271
2780099118,2cyclictest23046-21ssh11:25:021
2780099118,2cyclictest21175-21cut08:00:221
2780099117,3cyclictest5142-21tr10:55:151
2780099117,3cyclictest3970-21df12:30:141
2780099117,3cyclictest17361-21ssh10:05:141
2780099117,3cyclictest16417-21cut11:40:151
2780099117,3cyclictest16152-21grep09:00:141
2780099113,7cyclictest0-21swapper/109:33:561
2779999119,1cyclictest2274-21sed07:20:260
2779999119,1cyclictest20138-21grep08:00:140
2779999119,1cyclictest10561-21grep08:45:260
2779999118,2cyclictest6583-21basename08:40:130
2779999118,2cyclictest5040-21ssh11:33:210
2779999118,2cyclictest4786-21tune2fs08:35:180
2779999118,2cyclictest32002-21ssh10:32:440
2779999118,2cyclictest3064-21basename07:25:140
2779999118,2cyclictest305-21basename07:20:010
2779999118,2cyclictest29020-21cut12:25:250
2779999118,2cyclictest25523-21date12:05:010
2779999118,2cyclictest23890-21grep08:05:270
2779999118,2cyclictest10262-21tr07:40:150
2779999110,10cyclictest9212-21wc07:35:260
2780399108,1cyclictest8337-21chrt08:41:133
2780399108,1cyclictest32242-21expr08:25:143
2780399108,1cyclictest23842sleep307:20:513
2780399108,1cyclictest168652sleep307:52:573
2780399107,2cyclictest9172-21grep08:45:153
2780399107,2cyclictest19357-21cat09:05:193
2780399107,2cyclictest10080-21taskset07:40:033
2780199109,1cyclictest14357-21grep08:55:172
2780199108,1cyclictest8515-21sed07:35:172
2780199108,1cyclictest16736-21grep09:00:162
2780199107,2cyclictest9181-21cut08:45:152
2780199107,2cyclictest28908-21chrt08:16:592
2780199107,2cyclictest17043-21apt07:55:002
2780099108,1cyclictest30260-21basename08:20:181
2780099108,1cyclictest29200-21sendmail-msp08:20:001
2780099108,1cyclictest28895-21cut07:10:161
2780099108,1cyclictest2579-21taskset07:23:231
2780099108,1cyclictest22778-21sed08:05:171
2780099108,1cyclictest10937-21chrt08:49:091
2780099107,2cyclictest8522-21chrt08:43:351
2780099107,2cyclictest31047-21expr07:15:141
2780099107,2cyclictest20536-21grep10:25:271
2780099107,2cyclictest15186-21ls07:50:151
2780099103,6cyclictest15807-21sendmail-msp09:00:011
2780099103,3cyclictest225-21systemd-journal07:45:001
2780099100,9cyclictest14658-21cat07:49:591
2779999108,2cyclictest18447-21ssh09:46:030
2779999108,1cyclictest24714-21cut08:10:140
2779999108,1cyclictest21474-21ssh11:05:000
2779999108,1cyclictest14657-21ssh11:20:080
2779999108,1cyclictest12406-21sed12:35:160
2779999108,1cyclictest11726-21ssh10:40:040
2779999108,1cyclictest10051-21ssh12:15:010
2779999107,2cyclictest6794-21ls11:15:140
2779999107,2cyclictest30206-21chrt07:13:110
2779999102,6cyclictest0-21swapper/012:20:230
278039997,1cyclictest12829-21expr07:45:133
278019997,1cyclictest361-21taskset08:25:182
278019991,2cyclictest225-21systemd-journal07:57:472
277999998,1cyclictest5900-21df_abs07:30:160
277999998,1cyclictest16450-21expr09:00:150
277999997,1cyclictest32515-21fw_packets08:25:160
277999997,1cyclictest14836-21taskset07:50:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional