You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 05:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 30, 2025 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3109229242,45sleep10-21swapper/119:05:421
3136229063,9sleep20-21swapper/219:09:122
3139728875,9sleep00-21swapper/019:09:380
3110428560,9sleep30-21swapper/319:05:523
3151999182,15cyclictest18100-21apt-get19:50:023
3151199180,17cyclictest0-21swapper/121:35:151
31509991814,3cyclictest13-21ksoftirqd/021:35:140
3151199160,0cyclictest0-21swapper/121:57:011
3151199151,1cyclictest24936-21grep20:00:251
31519991411,2cyclictest13703-21head23:00:173
31519991410,3cyclictest23045-21cron20:00:003
31519991410,3cyclictest18342-21wc22:00:263
31515991411,2cyclictest29068-21turbostat.cron00:40:002
31515991410,3cyclictest21818-21grep23:15:282
31515991410,3cyclictest17588-21grep19:45:252
31515991410,2cyclictest11455-21tr19:35:172
3151199149,3cyclictest14287-21cat21:55:011
3151199142,8cyclictest27054-21apache220:05:141
31511991410,3cyclictest19956-21sed22:05:191
3150999149,2cyclictest21659-21tail19:55:170
3150999147,5cyclictest13-21ksoftirqd/021:15:190
31509991410,2cyclictest17426-21cut22:00:170
31509991410,2cyclictest13494-21cat19:40:000
3150999140,7cyclictest0-21swapper/022:35:140
3151999139,3cyclictest27492-21sed20:05:273
3151999139,3cyclictest11706-21wc20:40:233
3151999139,2cyclictest5179-21tr21:35:183
31519991311,1cyclictest28152-21apt-get20:10:153
31519991310,2cyclictest31848-21grep22:30:183
31519991310,2cyclictest26993-21latency_hist23:30:003
31519991310,2cyclictest23764-21cat22:15:003
31519991310,2cyclictest17762-21sed00:15:193
3151599139,3cyclictest8086-21grep19:25:282
3151599139,3cyclictest6449-21cut20:30:152
31515991310,2cyclictest19864-21perf23:15:002
31515991310,2cyclictest15240-21sort00:10:182
31515991310,2cyclictest12209-21grep22:55:282
31515991310,2cyclictest11193-21grep20:40:182
3151599130,2cyclictest0-21swapper/223:30:162
3151599130,12cyclictest0-21swapper/223:40:142
3151199139,3cyclictest27604-21tr21:15:151
3151199139,3cyclictest21786-21tr22:10:151
3151199139,3cyclictest21216-21grep19:55:151
3151199139,3cyclictest15942-21cut20:50:171
3151199131,7cyclictest38150irq/131-snd_hda00:05:141
31511991310,2cyclictest24542-21kworker/1:123:25:201
3150999139,3cyclictest23276-21sed23:20:180
3150999139,3cyclictest2007-21sleep23:45:000
3150999139,3cyclictest18082-21cron19:50:000
3150999138,3cyclictest4471-21wc22:40:190
3150999133,9cyclictest32682-21sendmail-msp20:20:000
31509991310,2cyclictest8956-21head22:50:170
31509991310,2cyclictest22241-21grep00:25:150
31509991310,2cyclictest18455-21tail23:10:170
31509991310,2cyclictest15961-21grep20:50:170
31509991310,2cyclictest15106-21dump-pmu-power23:05:000
31509991310,2cyclictest13694-21grep21:50:250
3151999129,2cyclictest8829-21grep23:55:293
3151999129,2cyclictest8594-21basename19:30:003
3151999129,2cyclictest7107-21grep22:45:223
3151999129,2cyclictest3667-21awk21:30:223
3151999129,2cyclictest29853-21grep22:25:213
3151999129,2cyclictest24210-21sed23:20:273
3151999129,2cyclictest2111-21cat21:30:003
3151999129,2cyclictest18436-21cut23:10:173
3151999128,3cyclictest2959-21wc19:15:233
3151999128,3cyclictest20828-21grep23:15:183
3151999128,3cyclictest19203-21sed20:55:273
3151999128,3cyclictest13048-21wc00:05:203
3151999128,2cyclictest598-21head23:40:173
3151999128,2cyclictest27103-21head22:20:193
3151999128,2cyclictest10704-21grep22:55:153
3151999128,2cyclictest10426-21grep21:45:153
3151999123,4cyclictest4642-21apt-get23:50:143
3151999123,4cyclictest4642-21apt-get23:50:143
3151999120,1cyclictest0-21swapper/322:50:143
3151599129,2cyclictest9557-21head19:30:182
3151599129,2cyclictest9337-21awk00:00:002
3151599129,2cyclictest5893-21wc23:50:222
3151599129,2cyclictest5893-21wc23:50:222
3151599129,2cyclictest4029-21df22:40:162
3151599129,2cyclictest30362-21cat20:15:002
3151599129,2cyclictest26600-21tr22:20:172
3151599129,2cyclictest24947-21sed20:00:252
3151599129,2cyclictest24637-21basename23:25:002
3151599129,2cyclictest24466-21awk21:10:002
3151599129,2cyclictest23651-21cron22:15:002
3151599129,2cyclictest14869-21wc19:40:212
3151599129,2cyclictest12539-21grep00:05:162
3151599129,2cyclictest12171-21grep21:50:142
3151599128,2cyclictest3771-21cut20:25:182
3151599123,2cyclictest0-21swapper/222:05:252
3151599120,11cyclictest0-21swapper/200:25:142
3151199129,2cyclictest20206-21grep19:50:271
3151199129,2cyclictest1752-21wc20:20:201
3151199129,2cyclictest15907-21wc19:45:001
3151199129,2cyclictest1553-21grep21:25:281
3151199129,2cyclictest15473-21tr23:05:141
3151199128,3cyclictest27798-21wc22:20:281
3151199128,3cyclictest1446-21perf19:15:011
3151199128,3cyclictest10646-21wc21:45:171
3151199128,2cyclictest9881-21grep00:00:151
3151199127,4cyclictest4747-21awk23:50:161
3151199127,4cyclictest4747-21awk23:50:151
3150999129,2cyclictest9886-21grep21:45:130
3150999129,2cyclictest30626-21head23:35:180
3150999129,2cyclictest30037-21cut21:20:130
3150999129,2cyclictest29847-21cat22:25:210
3150999129,2cyclictest28553-21cat22:25:000
3150999129,2cyclictest26045-21ls22:20:000
3150999129,2cyclictest25998-21cut20:05:160
3150999129,2cyclictest20840-21grep22:05:290
3150999129,2cyclictest16085-21grep23:05:170
3150999129,2cyclictest15858-21grep00:10:230
3150999128,3cyclictest4627-21apt-config21:35:000
3150999128,3cyclictest14661-21tr21:55:160
3150999125,3cyclictest225-21systemd-journal20:25:000
3150999121,6cyclictest29651-21munin-run23:35:000
3151999118,2cyclictest7714-21expr21:40:153
3151999118,2cyclictest4026-21chrt23:45:433
3151999118,2cyclictest3717-21grep22:40:163
3151999118,2cyclictest32680-21awk20:20:013
3151999118,2cyclictest28184-21grep00:35:253
3151999118,2cyclictest26783-21chrt21:11:533
3151999118,2cyclictest22629-21cut21:05:173
3151999118,2cyclictest21220-21wc21:00:253
3151999118,2cyclictest17389-21cat20:55:003
3151999118,2cyclictest15368-21apache_volume23:05:123
3151999118,2cyclictest13371-21sendmail-msp19:40:003
3151999117,3cyclictest3719-21grep20:25:183
3151999117,3cyclictest21128-21grep00:20:283
3151999114,2cyclictest15710-21cron19:45:003
3151999112,4cyclictest25293-21gltestperf00:30:193
3151599119,1cyclictest26229-21cut21:10:212
3151599118,2cyclictest9882-21tr00:00:152
3151599118,2cyclictest9874-21sed22:50:272
3151599118,2cyclictest7525-21sed22:45:262
3151599118,2cyclictest524-21grep21:25:192
3151599118,2cyclictest4368-21tr19:20:162
3151599118,2cyclictest3865-21grep23:45:252
3151599118,2cyclictest32039-21cat19:10:162
3151599118,2cyclictest315-21wc20:20:002
3151599118,2cyclictest31517-21taskset22:30:162
3151599118,2cyclictest30645-21head23:35:182
3151599118,2cyclictest29863-21tr21:20:122
3151599118,2cyclictest27652-21tr21:15:152
3151599118,2cyclictest25604-21sed22:15:252
3151599118,2cyclictest25261-21sort00:30:182
3151599118,2cyclictest21970-21chrt21:03:512
3151599118,2cyclictest18972-21taskset00:18:132
3151599118,2cyclictest17209-21taskset23:06:442
3151599118,2cyclictest17060-21grep22:00:152
3151599118,2cyclictest16514-21apt22:00:002
3151599118,2cyclictest1430-21head20:20:172
3151599118,2cyclictest11783-21munin-run21:50:002
3151599117,3cyclictest6999-21apt21:40:002
3151599117,3cyclictest19692-21sleep21:00:002
3151599116,3cyclictest8303-21grep20:35:162
3151599115,4cyclictest18083-21cron19:50:002
31515991110,1cyclictest0-21swapper/221:30:132
3151599110,10cyclictest17190-21seq20:53:432
3151199119,1cyclictest28182-21grep20:10:151
3151199118,2cyclictest8975-21uname20:35:201
3151199118,2cyclictest8115-21sed19:25:281
3151199118,2cyclictest31371-21grep21:20:251
3151199118,2cyclictest31141-21tr23:35:221
3151199118,2cyclictest3041-21head21:30:171
3151199118,2cyclictest27572-21tr23:30:161
3151199118,2cyclictest22549-21grep23:20:151
3151199118,2cyclictest21517-21sh00:25:001
3151199118,2cyclictest17711-21grep20:55:151
3151199118,2cyclictest17647-21cut00:15:191
3151199118,2cyclictest17245-21wc19:45:211
3151199118,2cyclictest17234-21expr22:00:161
3151199118,2cyclictest13738-21tr23:00:171
3151199118,2cyclictest1331-21cat23:40:261
3151199117,3cyclictest7903-21taskset22:49:251
3151199117,3cyclictest32273-21tr22:30:221
3151199117,1cyclictest25794-21taskset22:17:051
3151199116,4cyclictest24056-21sed21:05:291
3151199113,5cyclictest3195-21cron22:39:591
3151199111,8cyclictest0-21swapper/121:00:171
3151199110,10cyclictest4734-21sed20:25:281
3150999118,2cyclictest8909-21sed23:55:290
3150999118,2cyclictest6447-21grep20:30:150
3150999118,2cyclictest31987-21tr19:10:160
3150999118,2cyclictest31355-21tr22:30:150
3150999118,2cyclictest29341-21ls20:10:230
3150999118,2cyclictest25769-21cat23:25:160
3150999118,2cyclictest25605-21memory00:30:210
3150999118,2cyclictest24723-21chrt20:00:220
3150999118,2cyclictest21935-21taskset23:16:270
3150999118,2cyclictest21797-21tr22:10:150
3150999118,2cyclictest20437-21expr21:00:190
3150999118,2cyclictest1687-21taskset21:25:350
3150999118,2cyclictest11906-21gzip22:55:230
3150999118,2cyclictest10965-21sed20:40:160
3150999117,3cyclictest18808-21cut19:50:140
3150999117,3cyclictest13247-21wc00:05:220
3150999117,2cyclictest27714-21sed00:35:210
3150999116,3cyclictest8293-21tr20:35:160
3150999111,9cyclictest8852-21apt-get19:30:140
187922110,1sleep10-21swapper/123:10:211
3151999109,1cyclictest39-21ksoftirqd/319:31:333
3151999108,1cyclictest8244-21egrep20:35:163
3151999108,1cyclictest31829-21taskset23:37:413
3151999108,1cyclictest317032sleep319:10:053
3151999108,1cyclictest27453-21taskset21:15:153
3151999108,1cyclictest23902-21chrt00:27:353
3151999108,1cyclictest20382-21cut22:05:243
3151999108,1cyclictest166142sleep300:13:283
3151999108,1cyclictest15102-21grep21:55:183
3151999107,2cyclictest9930-21wc00:00:153
3151999107,2cyclictest4383-21expr19:20:163
3151999107,2cyclictest30203-21expr21:20:133
3151999107,2cyclictest25211-21taskset20:02:063
3151999107,2cyclictest2028-21gzip20:20:223
3151999107,2cyclictest17755-21taskset19:45:483
3151999107,2cyclictest13310-21df20:45:173
3151999107,2cyclictest12742-21head21:50:163
3151999101,8cyclictest6829-21ntp_states20:30:193
3151999101,7cyclictest27540-21apt-get23:30:153
3151599108,1cyclictest30109-21sendmail_mailtr22:25:232
3151599108,1cyclictest27762-21chrt20:08:382
3151599108,1cyclictest2504-21hostname19:15:192
3151599107,2cyclictest7966-21cut21:40:172
3151599107,2cyclictest26543-21grep23:25:232
3151599107,2cyclictest20127-21head00:20:192
3151599107,2cyclictest1033-21chrt22:35:022
3151599105,4cyclictest22353-21taskset19:55:252
3151199108,1cyclictest9118-21chrt23:57:271
3151199108,1cyclictest5168-21cat19:20:221
3151199108,1cyclictest287672sleep100:36:221
3151199108,1cyclictest2705-21cat23:45:141
3151199108,1cyclictest25518-21forks21:10:161
3151199108,1cyclictest25047-21expr00:30:171
3151199108,1cyclictest15028-21cat00:10:161
3151199108,1cyclictest130112sleep119:36:071
3151199108,1cyclictest11744-21ntpq22:55:221
3151199107,2cyclictest6389-21grep20:30:161
3151199107,2cyclictest5033-21cat22:40:261
3151199107,2cyclictest31543-21load20:15:201
3151199107,2cyclictest29379-21uniq22:25:171
3151199107,2cyclictest22089-21cut00:25:141
3151199107,2cyclictest11039-21forks20:40:171
3151199104,3cyclictest9079-21cut19:30:151
3151199102,4cyclictest225-21systemd-journal20:50:001
3151199102,1cyclictest0-21swapper/122:50:141
3151199100,9cyclictest90892sleep121:41:121
3150999108,2cyclictest13-21ksoftirqd/023:45:130
3150999108,1cyclictest1853-21grep19:15:160
3150999107,2cyclictest6333-21taskset22:45:150
3150999107,2cyclictest6273-21grep23:50:270
3150999107,2cyclictest6273-21grep23:50:270
3150999107,2cyclictest6065-21chrt19:23:200
3150999107,2cyclictest5156-21taskset20:29:340
3150999107,2cyclictest25116-21egrep21:10:140
3150999107,2cyclictest24448-21dump-pmu-power21:10:000
3150999107,2cyclictest21508-21cron00:25:000
3150999107,2cyclictest18661-21cat20:55:200
3150999102,4cyclictest19139-21dump-pmu-power00:20:000
3150999100,9cyclictest15601-21taskset19:43:360
3150999100,9cyclictest14139-21grep20:45:250
3150999100,9cyclictest10726-21cut00:00:220
315199997,1cyclictest2389-21uname22:35:233
315199996,2cyclictest24520-21sort22:15:153
315159996,2cyclictest13752-21head23:00:172
315159996,2cyclictest13590-21grep20:45:192
315119997,1cyclictest2158-21expr19:15:161
315119997,1cyclictest20560-21cat23:15:161
315099997,1cyclictest64272sleep019:25:010
315099994,3cyclictest8078-21hddtemp_smartct21:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional