You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 20:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Aug 26, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
486829952,43sleep00-21swapper/007:08:180
491529279,9sleep20-21swapper/207:08:562
489628772,11sleep30-21swapper/307:08:403
500028568,13sleep10-21swapper/107:10:001
509299281,1cyclictest0-21swapper/111:10:261
509299252,22cyclictest0-21swapper/108:30:161
509299250,23cyclictest0-21swapper/109:16:111
509799241,22cyclictest0-21swapper/309:50:133
509499234,3cyclictest141rcu_preempt10:00:592
509299230,3cyclictest0-21swapper/112:28:241
509299220,1cyclictest0-21swapper/109:55:381
509799210,1cyclictest0-21swapper/309:48:433
509499212,17cyclictest13223-21apt-get10:55:132
509499210,2cyclictest31148-21kworker/u8:009:41:572
509299210,0cyclictest0-21swapper/110:36:281
509299200,16cyclictest0-21swapper/110:58:301
508799200,2cyclictest0-21swapper/011:35:380
5094991917,1cyclictest0-21swapper/211:00:322
509499191,17cyclictest7642-21ssh12:09:132
509299191,1cyclictest2040-21cstates08:10:151
5087991917,1cyclictest2233-21ssh10:10:130
508799190,1cyclictest141rcu_preempt11:30:040
5097991816,1cyclictest6573-21ssh11:10:143
509499183,14cyclictest0-21swapper/208:32:122
509499180,1cyclictest0-21swapper/209:55:532
509299180,17cyclictest0-21swapper/110:25:201
509299180,17cyclictest0-21swapper/109:36:261
508799183,7cyclictest22945-21munin-run08:55:000
509499175,3cyclictest225-21systemd-journal09:05:012
509499170,17cyclictest0-21swapper/211:50:122
5092991714,2cyclictest30309-21diskmemload11:36:031
5094991615,1cyclictest0-21swapper/210:17:432
5094991615,1cyclictest0-21swapper/209:50:332
509499160,1cyclictest0-21swapper/210:35:022
509499160,15cyclictest0-21swapper/211:30:222
5092991615,1cyclictest30309-21diskmemload11:56:351
509299160,15cyclictest13108-21kworker/u8:111:55:011
508799161,11cyclictest8243-21apache_processe11:50:140
509799151,13cyclictest30309-21diskmemload12:15:293
5094991514,1cyclictest0-21swapper/210:47:532
5094991510,3cyclictest29504-21grep08:00:132
509499150,13cyclictest0-21swapper/210:23:512
509299152,10cyclictest32377-21ssh11:45:141
509299151,1cyclictest0-21swapper/111:42:371
509299150,15cyclictest0-21swapper/111:26:441
509299150,14cyclictest0-21swapper/112:30:141
509299150,14cyclictest0-21swapper/107:20:121
5087991511,3cyclictest13-21ksoftirqd/010:36:300
508799150,2cyclictest0-21swapper/009:16:080
508799150,15cyclictest0-21swapper/011:44:520
243532150,2sleep00-21swapper/011:00:590
509799149,3cyclictest18194-21ls12:35:003
509799141,1cyclictest141rcu_preempt11:32:113
5097991410,3cyclictest22801-21df07:45:153
509799140,13cyclictest8801-21cat08:25:003
509499148,6cyclictest0-21swapper/207:45:132
509499148,5cyclictest6794-21tail10:50:272
509499141,4cyclictest0-21swapper/211:15:132
5094991414,0cyclictest0-21swapper/210:33:432
5094991413,1cyclictest32-21ksoftirqd/209:27:032
5094991411,2cyclictest28526-21chrt10:25:182
5094991410,3cyclictest7200-21grep07:10:262
5092991414,0cyclictest0-21swapper/107:50:141
5092991412,1cyclictest1014-21grep08:05:261
509299141,1cyclictest0-21swapper/112:39:091
5092991411,2cyclictest2846-21tail09:50:251
5092991410,3cyclictest2743-21grep12:05:261
509299140,13cyclictest9521-21grep07:15:261
509299140,13cyclictest0-21swapper/111:00:211
508799148,4cyclictest7731-21munin-run11:50:010
508799141,7cyclictest31624-21awk09:30:000
508799141,7cyclictest17268-21basename10:00:000
5087991410,3cyclictest26276-21tr12:20:160
5087991410,2cyclictest1984-21tr08:10:150
508799140,3cyclictest0-21swapper/012:34:380
266932140,1sleep226694-21cat09:45:272
509799139,3cyclictest533-21cut08:05:203
509799139,3cyclictest19748-21sed10:00:263
509799139,3cyclictest16164-21cat09:40:003
509799139,2cyclictest24734-21grep11:20:263
509799139,2cyclictest0-21swapper/310:53:493
509799138,1cyclictest0-21swapper/309:26:473
509799131,9cyclictest30672-21rm10:26:123
5097991310,2cyclictest6746-21ls07:10:223
5097991310,2cyclictest32674-21grep09:10:253
5097991310,2cyclictest2318-21tr12:25:123
5097991310,2cyclictest17849-21tr09:40:213
509799130,13cyclictest0-21swapper/311:47:173
509799130,12cyclictest4346-21munin-plugin-st10:50:013
509499139,2cyclictest6318-21tr11:10:142
509499138,3cyclictest10631-21tr12:30:142
5094991310,2cyclictest3850-21turbostat08:15:002
5094991310,2cyclictest26721-21date12:40:002
5094991310,2cyclictest19518-21awk08:45:192
5094991310,2cyclictest0-21swapper/212:00:232
5094991310,2cyclictest0-21swapper/207:50:172
509499130,13cyclictest0-21swapper/207:43:252
509499130,12cyclictest786-21cut11:45:172
509299139,3cyclictest29058-21cut10:45:161
509299139,3cyclictest26673-21ls12:00:241
509299139,3cyclictest20471-21grep10:20:201
509299139,3cyclictest17274-21wc07:35:001
509299133,8cyclictest12710-21cron10:55:011
509299133,6cyclictest1142-21runrttasks09:13:161
509299131,6cyclictest22858-21kworker/1:009:25:141
5092991310,2cyclictest9328-21grep12:10:141
5092991310,2cyclictest23127-21ls08:55:011
5092991310,2cyclictest19555-21sh07:40:001
5092991310,2cyclictest11885-21tr10:15:201
509299130,2cyclictest0-21swapper/111:22:311
509299130,13cyclictest0-21swapper/111:05:281
509299130,12cyclictest0-21swapper/112:16:011
508799139,3cyclictest31015-21grep08:00:250
508799139,3cyclictest21509-21sed07:40:260
508799139,2cyclictest29123-21ls08:00:000
508799137,5cyclictest181irq_work/009:20:130
508799131,6cyclictest8852-21munin-run09:54:590
5087991311,2cyclictest0-21swapper/009:42:590
45702130,0chrt0-21swapper/110:10:421
509799129,2cyclictest8220-21head09:15:223
509799129,2cyclictest3134-21sort10:10:173
509799129,2cyclictest26953-21tr09:00:203
509799129,2cyclictest26021-21cut12:20:153
509799129,2cyclictest24433-21munin-plugin-st12:00:013
509799129,2cyclictest22529-21sed11:00:183
509799129,2cyclictest20065-21sed08:45:243
509799129,2cyclictest12010-21head08:30:163
509799128,3cyclictest39-21ksoftirqd/308:20:003
509799126,5cyclictest14363-21apt-get11:15:133
509799126,3cyclictest0-21swapper/310:23:223
5097991211,1cyclictest39-21ksoftirqd/311:54:013
509499129,2cyclictest914-21uname09:30:192
509499129,2cyclictest9004-21grep09:35:182
509499129,2cyclictest7219-21grep08:20:172
509499129,2cyclictest668-21grep11:25:242
509499129,2cyclictest2940-21grep10:10:162
509499129,2cyclictest23323-21wc09:25:002
509499129,2cyclictest19531-21cron07:40:002
509499129,2cyclictest19307-21ssh12:15:262
509499129,2cyclictest16826-21cut11:55:142
509499129,2cyclictest14246-21cat07:25:252
509499128,3cyclictest5211-21grep08:15:202
509499128,3cyclictest30856-21ssh09:10:132
509499122,9cyclictest56550irq/124-enp0s3107:55:152
5094991212,0cyclictest0-21swapper/211:42:032
5094991211,1cyclictest0-21swapper/211:39:432
509499120,6cyclictest28200-21unixbench_multi10:05:272
509499120,0cyclictest0-21swapper/210:42:532
509299129,2cyclictest9663-21grep08:25:171
509299129,2cyclictest24381-21sleep07:50:001
509299129,2cyclictest17720-21grep08:40:261
509299129,2cyclictest17286-21grep09:40:171
509299128,3cyclictest8119-21grep08:20:251
508799129,2cyclictest9869-21dump-pmu-power12:30:000
508799129,2cyclictest7618-21ls09:35:000
508799129,2cyclictest6078-21grep10:50:190
508799129,2cyclictest28228-21sh10:25:160
508799129,2cyclictest18629-21grep10:00:170
508799129,2cyclictest17532-21grep07:35:130
508799129,2cyclictest14277-21sort08:35:180
508799129,2cyclictest14277-21sort08:35:180
508799129,2cyclictest10821-21wc12:10:260
508799128,3cyclictest18489-21cron10:20:000
508799128,2cyclictest28177-21grep10:05:270
508799120,1cyclictest0-21swapper/011:19:110
508799120,11cyclictest25778-21head09:45:180
509799119,1cyclictest30045-21taskset09:06:563
509799118,2cyclictest9276-21ssh09:55:103
509799118,2cyclictest8579-21ssh12:09:573
509799118,2cyclictest3448-21ssh10:30:043
509799118,2cyclictest32498-21cat09:30:133
509799118,2cyclictest30318-21grep11:05:153
509799118,2cyclictest27279-21grep07:55:143
509799118,2cyclictest2591-21ssh11:26:503
509799118,2cyclictest25360-21head07:50:173
509799118,2cyclictest25080-21cut11:40:193
509799118,2cyclictest24465-21ssh10:41:563
509799118,2cyclictest2185-21expr08:10:153
509799118,2cyclictest20203-21ssh12:35:223
509799118,2cyclictest19448-21taskset07:39:013
509799118,2cyclictest18901-21ssh11:36:283
509799118,2cyclictest13691-21ssh10:35:233
509799118,2cyclictest13190-21apache_accesses10:55:113
509799117,3cyclictest8949-21grep08:25:133
509799117,3cyclictest22492-21sed08:50:273
509799117,3cyclictest11473-21head10:15:173
509799117,3cyclictest10015-21sed12:10:173
509799116,2cyclictest225-21systemd-journal09:00:003
509799114,6cyclictest24918-21ssh12:00:123
509799110,11cyclictest0-21swapper/310:07:213
509499118,2cyclictest9699-21ssh09:16:082
509499118,2cyclictest9573-21sed07:15:262
509499118,2cyclictest9184-21ssh12:10:142
509499118,2cyclictest2827-21ssh11:07:552
509499118,2cyclictest25780-21ssh12:20:082
509499118,2cyclictest24348-21tr11:20:222
509499117,3cyclictest12012-21grep07:20:262
509499113,5cyclictest16124-21grep08:40:132
509499110,10cyclictest4609-21ssh12:25:512
509299118,2cyclictest9300-21ssh10:33:161
509299118,2cyclictest4561-21expr08:15:151
509299118,2cyclictest32457-21ssh12:23:591
509299118,2cyclictest32379-21expr09:30:131
509299118,2cyclictest31716-21cat08:05:001
509299118,2cyclictest29159-21sendmail-msp07:59:591
509299118,2cyclictest27541-21tr10:05:211
509299118,2cyclictest26372-21cut09:45:231
509299118,2cyclictest22179-21ls10:40:251
509299118,2cyclictest21551-21ssh10:01:461
509299118,2cyclictest20013-21grep08:45:241
509299118,2cyclictest19877-21ssh09:22:281
509299118,2cyclictest14897-21df11:15:151
509299118,2cyclictest13580-21cat07:25:181
509299118,2cyclictest11380-21sh11:32:081
508799118,2cyclictest935-21ssh11:06:260
508799118,2cyclictest8663-21grep07:15:170
508799118,2cyclictest7655-21awk07:15:010
508799118,2cyclictest7319-21ssh10:31:430
508799118,2cyclictest6236-21cpu11:10:140
508799118,2cyclictest30889-21grep10:45:290
508799118,2cyclictest29050-21if_enp2s009:05:170
508799118,2cyclictest27593-21ssh11:22:360
508799118,2cyclictest26926-21ssh12:00:260
508799118,2cyclictest24154-21cut08:55:200
508799118,2cyclictest21602-21ssh12:17:030
508799118,2cyclictest21297-21ssh12:35:520
508799118,2cyclictest19334-21ssh10:58:390
508799118,2cyclictest19074-21ssh10:20:070
508799118,2cyclictest18541-21wc08:45:130
508799118,2cyclictest17156-21cron07:35:000
508799118,2cyclictest13942-21uname07:25:210
508799118,2cyclictest11810-21wc07:20:250
508799113,4cyclictest225-21systemd-journal09:00:030
508799110,10cyclictest7863-21ssh09:35:020
508799110,10cyclictest32272-21grep09:10:210
508799110,10cyclictest23578-21ssh10:41:160
509799108,1cyclictest217212sleep307:41:423
509799108,1cyclictest16926-21cat08:40:183
509799108,1cyclictest16254-21cut07:30:213
509799108,1cyclictest15258-21ssh09:20:123
509799108,1cyclictest14055-21expr08:35:163
509799108,1cyclictest14055-21expr08:35:163
509799108,1cyclictest12988-21cat07:25:143
509799107,2cyclictest9761-21taskset07:16:533
509799107,2cyclictest30465-21sed08:00:193
509799107,2cyclictest11960-21sed07:20:253
509499108,1cyclictest9428-21sed08:25:152
509499108,1cyclictest23565-21expr08:55:162
509499108,1cyclictest15702-21taskset08:39:452
509499108,1cyclictest15702-21taskset08:39:452
509499107,2cyclictest32029-21grep08:05:142
509499107,2cyclictest30489-21cat09:10:012
509499107,2cyclictest21551-21expr08:50:162
509299108,1cyclictest29109-21uname09:05:181
509299108,1cyclictest25773-21chrt09:00:051
509299108,1cyclictest25160-21chrt08:58:131
509299108,1cyclictest19870-21cut07:40:141
509299107,2cyclictest5540-21aten2_r7power_c07:10:141
509299107,2cyclictest15668-21chrt08:39:181
509299107,2cyclictest15668-21chrt08:39:181
508799108,1cyclictest31833-21aten2_r7power_e08:05:140
508799108,1cyclictest26153-21cat07:50:260
508799108,1cyclictest24214-21chrt07:48:110
508799108,1cyclictest18696-21ssh11:55:340
508799107,2cyclictest6742-21grep08:20:140
508799107,2cyclictest12125-21cat08:30:170
508799106,3cyclictest2789-21grep12:05:250
508799100,9cyclictest32116-21ssh11:25:160
50949997,1cyclictest16028-21cat07:30:182
50879997,1cyclictest5114-21cat08:15:190
50879997,1cyclictest160762sleep008:40:120
50879996,2cyclictest9479-21grep08:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional