You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 10:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jan 03, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
532529381,8sleep00-21swapper/019:09:430
518829166,9sleep20-21swapper/219:07:562
469429167,18sleep30-21swapper/319:05:283
510428265,13sleep10-21swapper/119:06:511
543599254,18cyclictest5746-21aten2_r7power_e19:10:151
543299198,8cyclictest537-21snmpd00:39:400
543599171,14cyclictest18479-21cat00:05:211
543899162,12cyclictest0-21swapper/223:15:142
543899161,1cyclictest0-21swapper/220:05:142
5435991615,0cyclictest0-21swapper/121:35:141
5432991612,3cyclictest13231-21cut23:55:150
544199152,5cyclictest0-21swapper/320:50:153
543299151,7cyclictest28057-21munin-run23:20:000
5432991511,3cyclictest30954-21wc20:00:250
543299150,7cyclictest0-21swapper/020:05:140
5438991410,2cyclictest20165-21sed20:45:262
543599147,6cyclictest27857-21head19:55:171
5435991411,2cyclictest13494-21cron20:35:001
5435991410,3cyclictest21226-21uname21:55:211
5435991410,2cyclictest19689-21munin-plugin-st19:40:001
543299145,6cyclictest161rcuc/020:25:160
543299142,6cyclictest31617-21turbostat.cron00:35:000
543299141,7cyclictest1613-21awk22:25:000
544199139,3cyclictest7173-21taskset21:25:263
544199139,3cyclictest10425-21tr23:50:143
544199139,2cyclictest24641-21wc22:05:013
5441991310,2cyclictest595-21cron21:15:003
5441991310,2cyclictest22162-21cat22:00:003
5441991310,2cyclictest19641-21grep20:45:203
5441991310,2cyclictest16276-21grep21:45:203
5441991310,2cyclictest14917-21latency_hist21:45:003
543899139,3cyclictest10222-21ls20:25:242
543899139,2cyclictest19947-21cat23:00:262
543899139,2cyclictest11471-21grep20:30:162
543899135,4cyclictest225-21systemd-journal00:35:002
5438991310,2cyclictest8597-21grep19:15:182
5438991310,2cyclictest31955-21wc21:10:192
5438991310,2cyclictest21585-21grep19:40:262
5438991310,2cyclictest21130-21grep00:10:222
5438991310,2cyclictest20688-21sed21:55:162
5438991310,2cyclictest17831-21tr19:35:152
5438991310,2cyclictest12617-21cat19:25:002
543599139,3cyclictest5229-21cat23:40:011
543599139,3cyclictest29991-21grep23:20:271
543599139,3cyclictest2759-21cron23:35:001
543599139,2cyclictest14504-21grep21:40:251
543599138,4cyclictest31154-21grep22:15:251
543599137,5cyclictest24639-21cat22:05:011
543599133,7cyclictest4873-21cut20:15:201
5435991310,2cyclictest32653-21head20:05:171
5435991310,2cyclictest27060-21grep21:00:211
5435991310,2cyclictest26476-21grep22:05:271
5435991310,2cyclictest22064-21sh19:45:001
5435991310,2cyclictest15282-21grep21:45:141
543599130,12cyclictest25550-21cut19:50:201
543299139,3cyclictest22947-21grep22:00:160
543299139,2cyclictest14943-21taskset19:30:000
543299138,4cyclictest24778-21apt-get19:50:160
543299135,7cyclictest23022-21munin-run23:10:000
543299133,9cyclictest13-21ksoftirqd/023:40:130
543299131,2cyclictest0-21swapper/000:05:140
5432991311,1cyclictest13-21ksoftirqd/023:50:120
5432991310,2cyclictest25981-21grep00:20:210
5432991310,2cyclictest18303-21grep19:35:170
5432991310,2cyclictest17657-21tr21:50:140
543299130,7cyclictest22070-21basename22:00:010
543299130,10cyclictest0-21swapper/022:05:150
544199129,2cyclictest9910-21cron23:50:013
544199129,2cyclictest4923-21grep20:15:203
544199129,2cyclictest3140-21tr23:35:153
544199129,2cyclictest30462-21dpkg21:10:003
544199129,2cyclictest30233-21head00:30:203
544199129,2cyclictest28468-21grep23:20:163
544199129,2cyclictest27915-21tail22:10:143
544199129,2cyclictest25376-21grep00:20:173
544199129,2cyclictest25319-21df19:50:183
544199129,2cyclictest23566-21tr23:10:143
544199129,2cyclictest2103-21tr22:25:143
544199129,2cyclictest21011-21grep23:05:143
544199129,2cyclictest13112-21head20:30:283
544199129,2cyclictest12699-21wc19:25:003
544199129,2cyclictest12168-21cut22:45:163
544199129,2cyclictest10693-21grep21:35:153
544199128,3cyclictest17609-21wc20:40:243
544199120,11cyclictest15931-21head19:30:173
544199120,11cyclictest0-21swapper/323:25:143
544199120,11cyclictest0-21swapper/323:25:143
543899129,2cyclictest7292-21grep22:35:172
543899129,2cyclictest4936-21grep21:20:262
543899129,2cyclictest32413-21sed23:25:272
543899129,2cyclictest32413-21sed23:25:262
543899129,2cyclictest2985-21cron21:20:002
543899129,2cyclictest29278-21cut23:20:202
543899129,2cyclictest27422-21chrt00:25:132
543899129,2cyclictest13845-21grep20:35:172
543899129,2cyclictest13564-21sed21:40:152
543899129,2cyclictest10906-21grep23:50:172
543899128,3cyclictest7779-21latency_hist21:30:002
543899128,3cyclictest15779-21sort21:45:172
543899128,2cyclictest24487-21basename00:20:002
543899126,2cyclictest225-21systemd-journal22:20:002
543899122,9cyclictest0-21swapper/220:20:142
543599129,2cyclictest9760-21sed19:15:281
543599129,2cyclictest681-21head00:35:211
543599129,2cyclictest5701-21cut21:25:141
543599129,2cyclictest1941-21cut20:10:161
543599129,2cyclictest16735-21sort22:55:171
543599129,2cyclictest10715-21cut19:20:181
543599128,3cyclictest9732-21tail22:40:181
543599128,3cyclictest28250-21wc23:20:011
543599128,3cyclictest24296-21taskset19:47:281
543599128,3cyclictest19643-21grep20:45:201
543599121,2cyclictest0-21swapper/122:35:141
543299129,2cyclictest49312sleep022:30:160
543299129,2cyclictest30013-21grep23:20:280
543299129,2cyclictest2354-21grep23:30:270
543299129,2cyclictest20682-21grep20:50:000
543299129,2cyclictest20017-21sort23:00:270
543299128,3cyclictest9613-21sort22:40:170
543299128,2cyclictest22350-21basename19:45:170
544199119,1cyclictest3364-21basename21:20:143
544199118,2cyclictest8305-21expr19:15:173
544199118,2cyclictest6365-21turbostat22:35:003
544199118,2cyclictest32177-21cat00:35:143
544199118,2cyclictest31317-21taskset22:16:213
544199118,2cyclictest29499-21taskset20:00:073
544199118,2cyclictest18431-21grep00:05:203
544199118,2cyclictest17798-21sed22:55:273
544199118,2cyclictest1747-21cat20:10:003
544199118,2cyclictest1526-21cut21:15:163
544199118,2cyclictest14408-21cut22:50:163
544199118,2cyclictest13281-21expr19:25:153
544199118,2cyclictest11280-21wc20:30:003
544199117,3cyclictest21619-21sed00:10:283
544199117,2cyclictest27011-21cat22:10:003
544199110,10cyclictest5995-21cat19:10:163
543899119,1cyclictest21172-21grep23:05:152
543899118,2cyclictest8584-21sed21:30:152
543899118,2cyclictest7196-21sed19:10:272
543899118,2cyclictest4983-21grep22:30:162
543899118,2cyclictest4734-21grep23:35:272
543899118,2cyclictest31614-21dump-pmu-power00:35:002
543899118,2cyclictest30406-21taskset21:09:432
543899118,2cyclictest2648-21grep20:10:182
543899118,2cyclictest25535-21grep00:20:182
543899118,2cyclictest25214-21expr19:50:182
543899118,2cyclictest25041-21grep22:05:162
543899118,2cyclictest24473-21tr23:10:192
543899118,2cyclictest23472-21chrt20:55:102
543899118,2cyclictest22739-21grep20:50:262
543899118,2cyclictest14882-21tr22:50:192
543899118,2cyclictest13561-21tr19:25:172
543899118,2cyclictest12135-21head22:45:162
543899117,3cyclictest22623-21tr19:45:182
543899113,5cyclictest18276-21awk20:45:002
543899110,10cyclictest18165-21cut21:50:162
543599118,2cyclictest8044-21taskset21:30:131
543599118,2cyclictest6889-21tr20:20:161
543599118,2cyclictest27799-21cut00:25:151
543599118,2cyclictest25360-21taskset23:13:591
543599118,2cyclictest25262-21head20:55:271
543599118,2cyclictest22743-21chrt23:06:381
543599118,2cyclictest20531-21sed00:10:171
543599118,2cyclictest19253-21grep23:00:171
543599118,2cyclictest18171-21grep21:50:161
543599118,2cyclictest16790-21sort20:40:151
543599118,2cyclictest14984-21head22:50:211
543599117,1cyclictest24394-21taskset00:19:141
543599113,6cyclictest733-21wc22:20:231
543299118,2cyclictest7294-21sort22:35:170
543299118,2cyclictest30724-21grep22:15:210
543299118,2cyclictest28077-21perf21:05:000
543299118,2cyclictest27773-21sort00:25:150
543299118,2cyclictest27414-21cut22:10:120
543299118,2cyclictest2731-21chrt21:16:470
543299118,2cyclictest25267-21head20:55:270
543299118,2cyclictest23065-21expr00:15:180
543299118,2cyclictest16985-21taskset00:04:110
543299118,2cyclictest14522-21grep20:35:200
543299118,2cyclictest12472-21taskset19:24:090
543299118,2cyclictest10513-21grep21:35:150
543299118,2cyclictest10139-21chrt19:19:560
543299114,4cyclictest23421-21aten2_r7power_v23:10:130
543299110,10cyclictest13-21ksoftirqd/021:45:130
544199108,1cyclictest8288-21expr21:30:153
544199108,1cyclictest32199-21expr22:20:163
544199108,1cyclictest29047-21taskset19:57:323
544199108,1cyclictest28030-21seq21:04:413
544199108,1cyclictest27440-21seq00:25:143
544199108,1cyclictest26352-21cat23:15:153
544199108,1cyclictest254002sleep320:56:393
544199108,1cyclictest2251-21sed23:30:263
544199108,1cyclictest20337-21expr19:40:153
544199108,1cyclictest19551-21memory23:00:203
544199108,1cyclictest15879-21cut00:00:193
544199108,1cyclictest10760-21taskset22:40:323
544199107,2cyclictest6909-21cut20:20:163
544199107,2cyclictest6598-21basename22:35:113
544199107,2cyclictest5881-21expr23:40:143
544199107,2cyclictest24190-21taskset19:46:083
544199107,2cyclictest22641-21grep00:15:163
544199107,2cyclictest2077-21grep20:10:163
544199107,2cyclictest17648-21cut21:50:143
544199107,2cyclictest17518-21chrt19:35:013
544199107,2cyclictest13292-21grep23:55:163
543899108,1cyclictest9466-21expr22:40:162
543899108,1cyclictest61112sleep220:16:552
543899108,1cyclictest57332sleep223:40:142
543899108,1cyclictest290322sleep219:57:202
543899108,1cyclictest28343-21grep22:10:182
543899108,1cyclictest27926-21taskset21:03:192
543899108,1cyclictest22650-21expr22:00:142
543899108,1cyclictest18320-21cat00:05:182
543899108,1cyclictest16519-21expr22:55:162
543899108,1cyclictest15531-21diskstats00:00:172
543899108,1cyclictest15192-21apache_volume19:30:132
543899107,2cyclictest7768-21taskset23:45:062
543899107,2cyclictest2916-21wc22:25:192
543899107,2cyclictest2764-21sh23:35:002
543899107,2cyclictest11122-21grep21:35:172
543899106,3cyclictest31209-21grep20:00:272
543899106,2cyclictest12591-21cut23:55:132
543599108,1cyclictest957-21cut21:15:141
543599108,1cyclictest8997-21open_inodes23:45:221
543599108,1cyclictest5383-21ntpq22:30:201
543599108,1cyclictest313522sleep100:31:501
543599108,1cyclictest31150-21cut21:10:141
543599108,1cyclictest3075-21grep22:25:201
543599108,1cyclictest25403-21uniq00:20:161
543599108,1cyclictest14338-21taskset23:55:581
543599108,1cyclictest14310-21cat19:25:251
543599108,1cyclictest13971-21wc20:35:171
543599108,1cyclictest10591-21expr23:50:151
543599108,1cyclictest10059-21expr20:25:211
543599107,2cyclictest4840-21cat21:20:261
543599107,2cyclictest30213-21cut20:00:171
543599107,2cyclictest28840-21sed22:10:231
543599107,2cyclictest14947-21chrt00:00:121
543599100,9cyclictest53892sleep123:40:041
543599100,9cyclictest32264-21chrt23:25:261
543599100,9cyclictest32264-21chrt23:25:261
543599100,9cyclictest301162sleep121:05:581
543599100,8cyclictest16281-21needreboot19:30:201
543299108,2cyclictest4422-21sensors_temp23:35:250
543299108,1cyclictest8777-21basename20:25:000
543299108,1cyclictest8427-21tune2fs23:45:170
543299108,1cyclictest4293-21basename20:15:170
543299108,1cyclictest3879-21fschecks_count21:20:160
543299108,1cyclictest30665-21find21:10:000
543299108,1cyclictest28394-21grep19:55:220
543299108,1cyclictest20202-21tr19:40:150
543299108,1cyclictest18058-21chrt22:57:490
543299108,1cyclictest15198-21dpkg19:30:130
543299108,1cyclictest13721-21uname21:40:170
543299107,2cyclictest9992-21chrt21:33:100
543299107,2cyclictest6520-21if_enp0s31f621:25:180
543299107,2cyclictest5781-21aten2_r7power_v19:10:150
543299107,2cyclictest31855-21chrt23:25:210
543299107,2cyclictest31855-21chrt23:25:210
543299107,2cyclictest31659-21head21:10:170
543299107,2cyclictest23019-21chrt20:52:190
543299107,2cyclictest2123-21expr22:25:140
543299107,2cyclictest16917-21expr20:40:150
543299107,2cyclictest11603-21cat20:30:170
543299107,2cyclictest11530-21egrep22:45:140
543299100,9cyclictest2984-21cut20:10:220
543299100,6cyclictest19942-21sshd00:10:100
54419997,1cyclictest14854-21sed20:35:213
54389997,1cyclictest32053-21grep22:20:162
54359994,4cyclictest11342-21chrt22:45:031
54359990,3cyclictest0-21swapper/120:55:001
54329997,1cyclictest15835-21taskset22:54:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional