You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 16:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Mar 24, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2944799571,2cyclictest0-21swapper/212:30:112
2944799511,46cyclictest0-21swapper/209:40:132
29457995019,19cyclictest28366irq/58-eth011:40:113
29447995026,17cyclictest0-21swapper/209:55:122
2944799493,43cyclictest22346irq/40-dwc2_hso07:45:122
2944799471,3cyclictest0-21swapper/208:20:142
2944799471,2cyclictest10500-21apt-get09:25:142
2944799471,2cyclictest0-21swapper/208:15:142
2942799472,7cyclictest131rcu_preempt09:45:140
29457994617,18cyclictest28366irq/58-eth010:35:143
2944799464,31cyclictest4236-21ssh11:45:122
2944799461,3cyclictest0-21swapper/208:25:132
2945799446,25cyclictest28366irq/58-eth011:55:133
2945799444,28cyclictest28366irq/58-eth012:10:133
2944799441,2cyclictest0-21swapper/210:30:132
2944799433,34cyclictest22346irq/40-dwc2_hso09:00:142
2944799423,35cyclictest1924-21apt-get07:15:142
2944799421,2cyclictest0-21swapper/212:05:122
2944799411,8cyclictest22699-21/usr/sbin/munin11:55:182
2944799411,6cyclictest18734-21kthreadcore09:10:232
2944799411,4cyclictest0-21swapper/211:25:152
29438994117,21cyclictest21-21ksoftirqd/110:25:121
29438994111,5cyclictest131rcu_preempt08:50:141
2945799403,22cyclictest28366irq/58-eth011:25:153
2944799402,7cyclictest0-21swapper/207:30:132
2944799402,2cyclictest0-21swapper/207:55:142
2944799402,2cyclictest0-21swapper/207:55:132
2944799401,4cyclictest0-21swapper/210:15:132
2944799401,4cyclictest0-21swapper/210:15:132
2944799401,3cyclictest0-21swapper/208:30:042
2944799401,28cyclictest0-21swapper/207:10:112
2943899402,27cyclictest17173-21ssh12:10:131
2945799394,22cyclictest28366irq/58-eth009:10:143
2945799391,23cyclictest28366irq/58-eth009:55:133
2944799393,28cyclictest9516-21ssh11:30:122
2944799392,6cyclictest0-21swapper/208:50:132
2944799391,6cyclictest29791-21grep11:05:192
2944799391,3cyclictest14823-21ntp_states10:55:272
2944799391,33cyclictest14842-21/usr/sbin/munin11:15:322
2944799390,5cyclictest11683-21sed12:25:012
2944799390,4cyclictest23146-21apt-get09:50:142
2942799398,4cyclictest131rcu_preempt08:00:130
2944799382,29cyclictest0-21swapper/211:50:082
2944799381,9cyclictest924-21snmpd10:38:472
2944799381,33cyclictest0-21swapper/208:00:122
2944799372,18cyclictest1119-21ssh12:35:242
2944799371,5cyclictest27555-21grep08:45:282
2944799371,3cyclictest0-21swapper/209:05:142
2944799371,33cyclictest16260-21missed_timers10:20:252
2944799371,32cyclictest20281-21fschecks_count11:00:172
2944799371,10cyclictest27162-21sendmail11:40:012
2944799363,2cyclictest19525-21apt-get09:30:142
2944799362,25cyclictest22344-21kthreadcore07:35:182
2944799362,17cyclictest0-21swapper/212:15:112
2944799361,32cyclictest14826-21irqstats08:35:182
2944799361,2cyclictest31419-21apt-get12:00:122
2944799360,5cyclictest5295-21kthreadcore10:50:242
2943899363,26cyclictest14189-21apt-get08:35:121
2945799354,19cyclictest28366irq/58-eth011:10:133
2944799352,3cyclictest0-21swapper/207:50:132
2944799352,3cyclictest0-21swapper/207:50:122
2944799351,7cyclictest14475-21sh09:45:162
2944799351,6cyclictest26317-21idleruntime07:40:162
2944799351,4cyclictest12372-21ntp_states10:00:272
2944799351,2cyclictest25412-21apt-get10:45:142
2944799350,6cyclictest11525-21sed07:25:022
2944799350,6cyclictest11525-21sed07:25:012
2942799356,4cyclictest12-21ksoftirqd/008:55:130
2942799351,4cyclictest131rcu_preempt09:00:140
2945799344,18cyclictest28366irq/58-eth011:20:113
2945799342,23cyclictest28366irq/58-eth010:05:143
2944799341,4cyclictest25647-21threads10:25:292
2944799341,3cyclictest0-21swapper/210:14:562
2944799341,2cyclictest0-21swapper/209:15:142
2944799341,2cyclictest0-21swapper/209:15:132
2944799340,6cyclictest17910-21ssh12:26:522
2945799334,16cyclictest28366irq/58-eth007:55:143
2945799334,16cyclictest28366irq/58-eth007:55:133
2944799333,26cyclictest5772-21munin-node11:10:132
2944799331,5cyclictest1433-21ssh09:20:092
2944799331,4cyclictest17162-21apt-get12:10:142
2944799331,2cyclictest0-21swapper/211:20:122
2944799331,2cyclictest0-21swapper/210:41:162
2944799330,5cyclictest22422-21latency_hist08:10:012
2945799321,18cyclictest28366irq/58-eth011:15:143
2944799326,23cyclictest18202-21cut10:05:172
2944799321,8cyclictest28525-21ssh09:35:072
2944799321,6cyclictest21860-21kthreadcore08:40:212
2944799320,4cyclictest29894-21grep11:40:182
2945799310,17cyclictest28366irq/58-eth012:20:133
2944799310,6cyclictest12231-21kernelversion07:25:182
2944799310,3cyclictest1062-21apt-get08:55:152
2942799315,23cyclictest13282-21ssh11:50:110
2945799302,16cyclictest28366irq/58-eth010:40:123
2944799309,19cyclictest22504-21/usr/sbin/munin08:10:172
2945799294,14cyclictest28366irq/58-eth012:35:123
2942799294,6cyclictest12-21ksoftirqd/012:30:110
2942799291,20cyclictest22346-21ssh11:55:140
2945799283,16cyclictest0-21swapper/312:25:233
2945799271,17cyclictest0-21swapper/310:50:143
2943899271,20cyclictest14200-21ssh09:45:141
2942799277,15cyclictest12-21ksoftirqd/010:40:130
2942799263,17cyclictest25443-21ssh10:45:160
2942799261,19cyclictest21718-21ssh10:25:130
2945799253,15cyclictest0-21swapper/312:30:173
2945799252,18cyclictest0-21swapper/310:15:143
2945799252,18cyclictest0-21swapper/310:15:133
2942799252,17cyclictest14885-21ssh11:15:140
2945799242,18cyclictest28366irq/58-eth010:45:173
2944799242,6cyclictest29477-21cat07:10:022
2942799247,9cyclictest22424-21kthreadcore07:35:180
2942799246,11cyclictest2899-21ssh12:20:140
2942799243,2cyclictest131rcu_preempt10:30:130
2945799235,10cyclictest28366irq/58-eth012:05:123
2945799233,14cyclictest28366irq/58-eth009:25:143
2943899237,8cyclictest0-21swapper/111:50:121
2943899235,10cyclictest26901-21ssh10:10:131
2943899233,14cyclictest0-21swapper/107:15:131
2943899231,13cyclictest26108-21irqrtprio10:45:221
2942799239,3cyclictest15232-21diskmemload11:30:420
2942799233,3cyclictest131rcu_preempt08:25:140
2942799231,16cyclictest5611-21ssh12:39:230
2942799231,16cyclictest19550-21ssh09:30:130
29427992310,9cyclictest0-21swapper/009:55:130
2945799224,9cyclictest321rcuc/308:25:143
2945799223,12cyclictest28366irq/58-eth011:45:113
29457992213,6cyclictest0-21swapper/311:09:173
2943899222,17cyclictest21-21ksoftirqd/109:50:131
2943899222,15cyclictest10521-21ssh09:25:151
2943899221,5cyclictest53146usb-storage10:20:041
29438992210,3cyclictest131rcu_preempt11:55:151
2942799225,3cyclictest131rcu_preempt11:45:140
2942799225,11cyclictest0-21swapper/009:25:140
2945799212,13cyclictest0-21swapper/311:35:163
2945799212,10cyclictest28366irq/58-eth008:00:133
29457992115,4cyclictest0-21swapper/310:55:203
29457992114,5cyclictest0-21swapper/312:15:243
29457992113,6cyclictest0-21swapper/311:50:243
29457992111,4cyclictest0-21swapper/309:15:183
29457992111,4cyclictest0-21swapper/309:15:173
29457992110,8cyclictest0-21swapper/310:32:103
2943899213,3cyclictest131rcu_preempt09:10:141
2943899213,13cyclictest0-21swapper/108:55:121
2943899211,17cyclictest31607-21cut12:00:171
2943899210,16cyclictest9696-21cut11:30:171
2942799213,5cyclictest12-21ksoftirqd/010:55:190
2942799212,15cyclictest11587-21apt-get07:25:120
2942799211,7cyclictest30127-21grep09:35:210
2942799211,16cyclictest23168-21ssh09:50:130
29427992111,3cyclictest15232-21diskmemload11:39:480
2945799202,11cyclictest0-21swapper/308:35:153
29457992014,4cyclictest0-21swapper/311:30:253
29457992014,4cyclictest0-21swapper/310:00:313
29457992014,4cyclictest0-21swapper/309:45:243
29457992014,4cyclictest0-21swapper/309:35:133
29457992014,4cyclictest0-21swapper/309:30:133
29457992014,4cyclictest0-21swapper/309:05:133
29457992014,4cyclictest0-21swapper/307:20:143
29457992014,4cyclictest0-21swapper/307:20:133
29457992013,5cyclictest321rcuc/309:50:123
29457992013,5cyclictest0-21swapper/310:10:123
29457992012,5cyclictest28366irq/58-eth012:00:143
29457992012,5cyclictest0-21swapper/307:45:283
2943899204,13cyclictest1634-21kthreadcore08:20:211
2943899203,15cyclictest4245-21ssh11:45:121
2943899202,12cyclictest19193-21ssh12:28:221
2943899201,16cyclictest9549-21rm11:10:281
2943899201,15cyclictest2455-21latency_hist12:20:011
2943899201,14cyclictest23575-21latency_hist08:45:021
2943899200,16cyclictest24094-21ls11:00:301
2942799202,15cyclictest15565-21ssh12:25:270
2942799201,2cyclictest131rcu_preempt12:05:120
2942799201,11cyclictest15232-21diskmemload10:52:140
2942799200,4cyclictest131rcu_preempt07:20:300
2942799200,4cyclictest131rcu_preempt07:20:290
2945799199,7cyclictest0-21swapper/310:25:133
2945799192,10cyclictest0-21swapper/308:50:043
29457991913,4cyclictest0-21swapper/311:00:233
29457991913,4cyclictest0-21swapper/310:20:033
29457991913,4cyclictest0-21swapper/309:41:203
29457991913,4cyclictest0-21swapper/309:20:233
29457991913,4cyclictest0-21swapper/307:15:123
2945799191,12cyclictest0-21swapper/307:30:153
2943899199,7cyclictest11473-21cron12:25:011
2943899196,10cyclictest5550-21kthreadcore08:25:201
2943899194,12cyclictest14561-21ssh10:55:241
2943899191,5cyclictest53146usb-storage08:30:111
29438991912,5cyclictest31251-21sh11:40:241
29438991912,5cyclictest0-21swapper/111:25:021
2943899191,14cyclictest18736-21fschecks_count11:35:171
2943899191,14cyclictest16355-21diskmemload10:40:121
2943899191,13cyclictest4494-21ls11:25:301
29438991910,6cyclictest20489-21cron12:30:031
2943899190,15cyclictest6504-21ls07:50:211
2943899190,15cyclictest6504-21ls07:50:211
2942799199,3cyclictest15232-21diskmemload10:06:060
2942799199,3cyclictest15232-21diskmemload09:13:480
2942799195,3cyclictest131rcu_preempt12:14:430
2942799192,11cyclictest31300-21ssh12:00:050
2942799192,10cyclictest0-21swapper/007:15:120
2942799191,9cyclictest28464-21cmp08:50:020
2942799191,4cyclictest0-21swapper/012:15:580
29427991911,6cyclictest2934-21sshd10:14:440
2942799191,15cyclictest5795-21apt-get11:10:130
2942799191,14cyclictest7455-21ssh10:35:150
29427991911,3cyclictest15232-21diskmemload10:18:300
29427991911,3cyclictest15232-21diskmemload10:18:290
2945799182,10cyclictest0-21swapper/308:20:133
2945799182,10cyclictest0-21swapper/307:40:053
29457991812,4cyclictest0-21swapper/308:55:173
29457991812,4cyclictest0-21swapper/307:10:153
29457991811,4cyclictest0-21swapper/308:30:173
2943899189,7cyclictest29131-21sed09:35:191
2943899189,6cyclictest18155-21sed10:05:181
2943899187,9cyclictest16481-21df_inode07:30:141
2943899183,4cyclictest21-21ksoftirqd/109:15:141
2943899183,4cyclictest21-21ksoftirqd/109:15:131
2943899182,13cyclictest3150-21memory09:55:241
2943899182,13cyclictest1519-21ssh09:20:131
2943899182,12cyclictest21-21ksoftirqd/109:30:121
2943899181,8cyclictest17697-21/usr/sbin/munin08:05:241
2943899181,7cyclictest2080-21ntp_states10:30:261
2943899181,13cyclictest14898-21sh11:15:151
2943899181,10cyclictest9169-21kthreadcore12:05:201
2942799189,3cyclictest15232-21diskmemload09:40:490
2942799188,8cyclictest30388-21tr11:05:220
2942799188,7cyclictest29852-21kthreadcore11:40:180
29427991812,4cyclictest0-21swapper/008:40:280
2942799181,15cyclictest663-21ssh11:25:170
2942799181,15cyclictest23701-21cut08:10:200
2942799181,13cyclictest24826-21ssh09:15:140
2942799181,13cyclictest24826-21ssh09:15:130
2942799181,11cyclictest31084-21timerwakeupswit08:15:310
29427991810,6cyclictest9899-21cut10:00:190
2945799172,10cyclictest0-21swapper/308:45:133
2945799172,10cyclictest0-21swapper/308:40:133
2945799172,10cyclictest0-21swapper/308:15:173
2945799172,10cyclictest0-21swapper/307:50:143
2945799172,10cyclictest0-21swapper/307:50:133
29457991712,3cyclictest0-21swapper/309:00:133
29457991711,4cyclictest0-21swapper/308:10:183
29457991711,4cyclictest0-21swapper/307:35:143
2943899179,6cyclictest12580-21sshd07:56:231
2943899179,6cyclictest12580-21sshd07:56:221
2943899179,6cyclictest12306-21vmstat10:35:361
2943899179,5cyclictest31148-21kthreadcore12:35:181
2943899178,7cyclictest12524-21tail09:05:201
29438991711,4cyclictest26355-21if_err_eth107:40:161
29438991711,4cyclictest0-21swapper/111:05:571
2943899171,13cyclictest5321-21diskstats09:40:151
2943899171,13cyclictest12615-21apt-get08:00:031
2943899170,8cyclictest916666ssh10:53:111
2942799179,6cyclictest618-21systemd-logind08:37:490
2942799179,6cyclictest21543-21cat11:00:200
2942799179,6cyclictest1-21systemd08:54:270
2942799178,7cyclictest5078-21ntp_192.168.11409:20:260
2942799177,3cyclictest15232-21diskmemload10:23:200
2942799174,2cyclictest131rcu_preempt07:30:240
2942799174,10cyclictest9686-21kthreadcore07:55:210
2942799174,10cyclictest9686-21kthreadcore07:55:200
2942799172,11cyclictest0-21swapper/008:30:100
29457991610,4cyclictest0-21swapper/308:05:153
29457991610,4cyclictest0-21swapper/307:25:243
2943899169,4cyclictest22672-21cut08:10:161
2943899163,10cyclictest710-21switchtime07:10:261
2943899162,11cyclictest3584-21ssh10:15:141
2943899162,11cyclictest3584-21ssh10:15:131
2943899162,11cyclictest31978-21kthreadcore07:45:191
2943899161,12cyclictest31030-21grep08:15:301
2943899161,12cyclictest21270-21diskstats07:35:151
2943899161,12cyclictest10186-21ntp_192.168.11407:20:241
2943899161,12cyclictest10186-21ntp_192.168.11407:20:231
29438991610,4cyclictest15232-21diskmemload10:00:131
2943899160,3cyclictest131rcu_preempt07:30:011
2942799169,5cyclictest3914-21usage_EthcTask08:20:300
2942799168,6cyclictest1669-21sshd07:11:230
2942799167,6cyclictest30814-21latency_hist07:45:020
2942799167,6cyclictest20968-21latency08:05:230
2942799161,13cyclictest6357-21kthreadcore07:50:220
2942799161,13cyclictest6357-21kthreadcore07:50:210
2942799161,12cyclictest24009-21df11:20:130
2942799160,13cyclictest32728-21kthreadcore07:45:190
2943899151,11cyclictest6996-21kthreadcore09:00:211
2943899151,11cyclictest23916-21fschecks_count08:45:141
2942799151,3cyclictest131rcu_preempt09:05:080
2943899133,8cyclictest29418-21cyclictest07:10:021
2942799136,5cyclictest29443-21latency_hist07:10:010
2945799101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional