You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 08:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Dec 17, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
715329265,9sleep00-21swapper/019:09:110
709928572,9sleep30-21swapper/319:08:283
691528270,8sleep20-21swapper/219:06:072
705527659,13sleep10-21swapper/119:07:551
7301992826,2cyclictest25-21ksoftirqd/100:07:571
7303992320,3cyclictest32-21ksoftirqd/200:07:572
730199200,17cyclictest6070-21ntp_states21:20:211
730199190,6cyclictest25-21ksoftirqd/121:55:201
730199190,6cyclictest25-21ksoftirqd/121:55:191
730199186,2cyclictest25-21ksoftirqd/123:37:001
730199180,12cyclictest0-21swapper/123:40:141
730399171,16cyclictest6110-21apt-get20:15:152
729299175,6cyclictest26533-21munin-run00:20:000
729299174,6cyclictest28701-21idleruntime-cro19:55:000
7303991614,1cyclictest24413-21apache_accesses00:15:102
730399160,14cyclictest0-21swapper/222:50:182
730399151,6cyclictest0-21swapper/219:40:142
7303991511,3cyclictest18908-21cron21:50:002
730199159,0cyclictest0-21swapper/120:15:151
729299151,7cyclictest14378-21munin-run23:55:000
7292991510,3cyclictest31742-21grep21:05:240
7307991410,3cyclictest4404-21grep22:25:183
730399142,11cyclictest0-21swapper/222:25:152
730399140,1cyclictest0-21swapper/222:05:152
730199147,7cyclictest0-21swapper/122:25:151
730199141,7cyclictest0-21swapper/119:45:141
730199141,5cyclictest0-21swapper/122:30:131
7301991410,3cyclictest16315-21grep19:25:291
729299142,6cyclictest26079-21perf22:05:010
7292991410,2cyclictest29433-21df19:55:150
730799139,3cyclictest830-21sed00:30:283
730799139,3cyclictest21849-21wc00:10:013
730799139,3cyclictest17740-21idleruntime-cro22:55:003
730799134,6cyclictest29862-21munin-run21:04:593
730799132,8cyclictest17605-21diskstats00:00:173
7307991310,2cyclictest9473-21sed20:20:173
7307991310,2cyclictest5450-21head20:10:283
7307991310,2cyclictest31398-21basename22:15:133
7307991310,2cyclictest2605-21grep00:35:203
7307991310,2cyclictest25174-21cut00:15:163
7307991310,2cyclictest13957-21grep19:20:263
7307991310,2cyclictest13708-21sed21:35:283
730799130,12cyclictest2358-21wc21:15:003
730399139,3cyclictest27697-21tr19:50:212
730399139,3cyclictest26453-21grep23:10:192
730399131,3cyclictest0-21swapper/221:20:142
730399131,1cyclictest0-21swapper/220:50:142
7303991310,2cyclictest586-21grep20:00:252
7303991310,2cyclictest586-21grep20:00:242
7303991310,2cyclictest26574-21grep20:55:222
7303991310,2cyclictest16949-21sort00:00:012
7303991310,2cyclictest16949-21sort00:00:012
7303991310,2cyclictest12550-21grep22:40:262
7303991310,2cyclictest11990-21sh23:50:002
7303991310,2cyclictest1034-21idleruntime-cro22:20:002
730399130,11cyclictest0-21swapper/221:00:132
730199139,3cyclictest27026-21grep23:10:251
730199139,3cyclictest26564-21tr00:20:011
730199138,4cyclictest14358-21cat20:30:181
730199136,5cyclictest16797-21sh22:50:211
730199135,7cyclictest24746-21hwlatdetect22:00:181
730199135,7cyclictest0-21swapper/123:55:161
730199135,7cyclictest0-21swapper/123:55:161
730199134,5cyclictest225-21systemd-journal21:10:001
7301991311,1cyclictest4888-21latency_hist23:35:001
7301991310,2cyclictest3614-21cron00:40:001
7301991310,2cyclictest27763-21basename21:00:151
7301991310,2cyclictest22022-21cat20:45:241
7301991310,2cyclictest18469-21grep20:40:181
7301991310,2cyclictest15527-21cut21:40:211
729299139,3cyclictest25606-21tr23:10:150
729299139,3cyclictest15946-21wc21:40:270
729299139,3cyclictest15451-21grep19:25:200
729299139,2cyclictest4456-21grep20:10:180
729299139,2cyclictest2750-21tr21:15:140
729299132,10cyclictest11763-21cron21:34:590
7292991310,2cyclictest5807-21sh22:30:010
7292991310,2cyclictest30507-21fgrep00:25:230
7292991310,2cyclictest29029-21cat00:25:010
7292991310,2cyclictest1022-21sh22:20:010
729299130,12cyclictest28729-21ls23:15:200
730799129,2cyclictest9772-21wc22:35:233
730799129,2cyclictest32249-21sleep21:10:003
730799129,2cyclictest29952-21sed00:25:173
730799129,2cyclictest27549-21head00:20:173
730799129,2cyclictest25090-21cron23:10:003
730799129,2cyclictest24210-21awk20:50:223
730799129,2cyclictest21328-21grep23:00:183
730799129,2cyclictest17531-21grep20:35:253
730799129,2cyclictest16612-21wc21:44:593
730799129,2cyclictest11549-21grep19:15:263
730799128,3cyclictest1615-21tr20:05:143
730799125,6cyclictest6077-21apt-get22:30:133
730799125,3cyclictest225-21systemd-journal20:30:013
730399129,2cyclictest3831-21wc00:40:002
730399129,2cyclictest3453-21cron20:10:012
730399129,2cyclictest31817-21grep21:05:252
730399129,2cyclictest29235-21tr19:55:142
730399129,2cyclictest25101-21dump-pmu-power23:10:002
730399129,2cyclictest23982-21grep22:00:142
730399129,2cyclictest16567-21head20:35:152
730399129,2cyclictest15919-21grep19:25:252
730399128,3cyclictest28413-21grep23:15:172
730399128,3cyclictest1588-21wc21:10:262
7303991210,1cyclictest22305-21head21:55:182
7303991210,1cyclictest22305-21head21:55:172
7303991210,1cyclictest0-21swapper/223:50:102
730199129,2cyclictest9712-21wc20:20:191
730199129,2cyclictest4854-21tr20:10:221
730199129,2cyclictest31550-21cat23:20:251
730199129,2cyclictest27064-21tr00:20:151
730199129,2cyclictest26924-21grep22:05:171
730199129,2cyclictest25-21ksoftirqd/121:40:001
730199129,2cyclictest24646-21tail23:05:261
730199129,2cyclictest17130-21cut19:30:131
730199129,2cyclictest12904-21grep23:50:151
730199128,3cyclictest14362-21grep22:45:191
730199123,4cyclictest225-21systemd-journal23:00:021
7301991211,1cyclictest0-21swapper/120:50:131
7301991210,2cyclictest25-21ksoftirqd/119:50:141
7301991210,1cyclictest18340-21cstates22:55:151
729299129,2cyclictest8526-21grep21:25:220
729299129,2cyclictest7167-21cat21:25:000
729299129,2cyclictest3426-21sh22:25:000
729299129,2cyclictest31754-21grep23:20:250
729299129,2cyclictest20105-21cron23:00:000
729299129,2cyclictest19471-21grep19:35:150
729299129,2cyclictest12008-21awk23:50:000
729299128,3cyclictest15671-21grep22:50:140
730799118,2cyclictest9271-21chrt21:28:293
730799118,2cyclictest7943-21chrt20:16:363
730799118,2cyclictest638-21sed20:00:253
730799118,2cyclictest638-21sed20:00:253
730799118,2cyclictest3783-21cat23:30:203
730799118,2cyclictest32626-21cut23:25:143
730799118,2cyclictest28465-21head23:15:183
730799118,2cyclictest25502-21grep23:10:133
730799118,2cyclictest22475-21taskset20:47:023
730799118,2cyclictest22232-21expr19:40:163
730799118,2cyclictest19984-21chrt20:40:323
730799118,2cyclictest12989-21head23:50:153
730799117,3cyclictest8304-21grep19:10:173
730799117,3cyclictest30825-21basename22:15:013
730799116,4cyclictest32235-21cron23:24:593
730799116,2cyclictest225-21systemd-journal21:19:593
730799111,2cyclictest225-21systemd-journal20:33:093
730799110,10cyclictest11914-21taskset23:49:093
730399118,2cyclictest9670-21taskset21:30:132
730399118,2cyclictest9208-21grep22:35:182
730399118,2cyclictest9123-21cat19:10:262
730399118,2cyclictest8114-21basename21:25:182
730399118,2cyclictest6884-21taskset23:35:532
730399118,2cyclictest6323-21grep22:30:142
730399118,2cyclictest4841-21uname20:10:222
730399118,2cyclictest3739-21grep21:15:202
730399118,2cyclictest30407-21head22:10:262
730399118,2cyclictest3020-21tr23:30:152
730399118,2cyclictest21011-21expr23:00:162
730399118,2cyclictest20862-21cat19:35:272
730399118,2cyclictest18791-21sed22:55:162
730399118,2cyclictest16154-21sed21:40:272
730399118,2cyclictest15644-21cat20:35:012
730399118,2cyclictest11243-21wc20:25:152
730399118,2cyclictest10548-21taskset20:21:302
730399117,3cyclictest11148-21grep19:15:212
730399117,2cyclictest0-21swapper/219:20:122
730399116,3cyclictest31569-21apt-get00:30:152
730399112,1cyclictest141rcu_preempt23:20:052
730199119,1cyclictest3015-21grep22:20:281
730199119,1cyclictest26086-21grep20:55:181
730199118,2cyclictest547-21tail22:15:261
730199118,2cyclictest3222-21grep21:15:151
730199118,2cyclictest30312-21tr00:25:201
730199118,2cyclictest28979-21grep22:10:151
730199118,2cyclictest28269-21sed23:15:161
730199118,2cyclictest25-21ksoftirqd/120:35:191
730199118,2cyclictest18361-21cat21:45:251
730199118,2cyclictest17096-21cut00:00:141
730199118,2cyclictest1626-21grep20:05:151
730199118,2cyclictest1236-21cron00:35:001
730199118,2cyclictest11888-21taskset19:18:241
730199118,2cyclictest10360-21expr23:45:151
730199115,2cyclictest225-21systemd-journal19:39:591
730199113,2cyclictest25-21ksoftirqd/121:26:051
730199110,7cyclictest225-21systemd-journal20:04:551
730199110,7cyclictest225-21systemd-journal20:04:541
730199110,6cyclictest25-21ksoftirqd/100:15:011
729299118,2cyclictest9814-21cat22:35:230
729299118,2cyclictest32743-21grep23:25:140
729299118,2cyclictest3262-21taskset20:07:350
729299118,2cyclictest32334-21sed00:30:190
729299118,2cyclictest30769-21chrt22:14:410
729299118,2cyclictest29670-21chrt21:02:450
729299118,2cyclictest276332sleep022:05:230
729299118,2cyclictest24984-21taskset20:53:480
729299118,2cyclictest23426-21sshd21:57:090
729299118,2cyclictest23426-21sshd21:57:090
729299118,2cyclictest21431-21systemd00:07:570
729299118,2cyclictest19344-21cat00:05:000
729299118,2cyclictest16755-21perf00:00:010
729299118,2cyclictest16755-21perf00:00:010
729299118,2cyclictest16601-21grep20:35:150
729299118,2cyclictest13324-21date20:30:010
729299118,2cyclictest12537-21cut19:20:130
729299117,3cyclictest8250-21tail23:40:160
729299116,3cyclictest21335-21head20:45:160
729299113,4cyclictest225-21systemd-journal20:00:010
729299113,4cyclictest225-21systemd-journal20:00:000
729299111,5cyclictest225-21systemd-journal21:55:000
729299110,10cyclictest22148-21cat23:00:260
730799108,1cyclictest9446-21taskset23:42:563
730799108,1cyclictest24891-21cut19:45:173
730799108,1cyclictest24759-21wc22:00:173
730799108,1cyclictest20334-21rm22:59:593
730799108,1cyclictest190122sleep319:33:083
730799108,1cyclictest16810-21apache_volume21:45:133
730799108,1cyclictest1640-21wc22:20:163
730799108,1cyclictest158432sleep319:25:243
730799108,1cyclictest13363-21cut22:45:143
730799107,2cyclictest5109-21apache_volume23:35:123
730799107,2cyclictest26655-21cut19:50:153
730799107,2cyclictest25438-21basename20:55:153
730799107,2cyclictest22784-21tail00:10:163
730799107,2cyclictest22101-21df21:55:163
730799107,2cyclictest22101-21df21:55:153
730799107,2cyclictest21281-21cron21:55:003
730799107,2cyclictest20036-21cut19:35:193
730799107,2cyclictest16444-21taskset23:56:233
730799107,2cyclictest16444-21taskset23:56:223
730799106,3cyclictest4951-21tr21:20:143
730799100,3cyclictest11079-21/usr/sbin/munin22:40:153
730399108,1cyclictest75202sleep223:40:142
730399108,1cyclictest28180-21readlink00:20:232
730399108,1cyclictest24392-21ls19:45:142
730399108,1cyclictest18415-21sendmail_mailtr19:30:222
730399108,1cyclictest13399-21wget22:45:132
730399107,2cyclictest23149-21strings00:10:192
730399107,2cyclictest21232-21cut20:45:152
730399107,2cyclictest20722-21cat21:50:272
730399107,2cyclictest18523-21grep20:40:182
730399106,4cyclictest311rcuc/221:36:432
730399106,2cyclictest29679-21df_abs00:25:152
730399105,4cyclictest32-21ksoftirqd/200:00:152
730399102,5cyclictest311rcuc/222:20:162
730199108,1cyclictest932-21hwlatdetect21:10:181
730199108,1cyclictest7865-21cat19:10:151
730199108,1cyclictest210892sleep121:52:351
730199108,1cyclictest1155-21cat23:25:181
730199108,1cyclictest11449-21cat20:25:151
730199108,1cyclictest11304-21cat22:40:151
730199107,2cyclictest9624-21chrt21:30:021
730199107,2cyclictest12980-21cut19:20:151
730199106,3cyclictest25-21ksoftirqd/122:35:141
730199103,5cyclictest25-21ksoftirqd/119:58:271
730199103,3cyclictest0-21swapper/119:40:141
729299108,1cyclictest9910-21apache_accesses19:15:120
729299108,1cyclictest8675-21seq20:20:020
729299108,1cyclictest8051-21taskset20:17:590
729299108,1cyclictest2991-21tr23:30:150
729299108,1cyclictest27193-21taskset20:56:420
729299108,1cyclictest21827-21apache_processe19:40:130
729299108,1cyclictest2056-21chrt21:12:390
729299108,1cyclictest1532-21cut00:35:140
729299108,1cyclictest14059-21smartctl22:45:160
729299107,2cyclictest9474-21taskset19:12:590
729299107,2cyclictest5855-21grep23:35:170
729299107,2cyclictest24863-21cut19:45:180
729299107,2cyclictest23736-21grep23:05:170
729299107,2cyclictest20019-21taskset20:40:580
729299107,2cyclictest17514-21grep21:45:160
729299107,2cyclictest13802-21expr20:30:150
729299105,4cyclictest11182-21cstates22:40:150
729299100,9cyclictest60342sleep022:30:080
73079997,1cyclictest28903-21chrt19:55:053
73079997,1cyclictest10754-21needreboot21:30:213
73039997,1cyclictest500-21cat23:25:152
72929997,1cyclictest23361-21sendmail_mailqu00:10:220
72929997,1cyclictest18550-21cat19:30:250
72929991,4cyclictest225-21systemd-journal21:35:010
73079986,1cyclictest281042sleep322:05:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional