You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 12:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Mar 29, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1157199733,48cyclictest31431-21ssh00:10:131
1156299543,38cyclictest2121-21ssh23:00:130
1156299511,41cyclictest1608-21ssh21:30:130
1158399471,28cyclictest23827-21ssh23:30:122
1159199462,24cyclictest28366irq/58-eth019:10:123
1157199450,35cyclictest23833-21ssh23:30:131
1159199445,24cyclictest28366irq/58-eth021:15:143
1156299441,36cyclictest22398-21ssh00:05:130
1158399431,34cyclictest1351-21kthreadcore19:30:202
11571994330,4cyclictest27132-21sh22:20:111
11571994214,21cyclictest21-21ksoftirqd/121:40:121
11562994222,12cyclictest12-21ksoftirqd/021:20:130
1159199404,21cyclictest28366irq/58-eth023:05:123
1159199403,20cyclictest28366irq/58-eth000:15:133
11591994019,18cyclictest28366irq/58-eth021:45:123
1158399401,28cyclictest437-21ssh23:35:142
1158399392,18cyclictest27522-21grep19:25:202
1157199395,25cyclictest29119-21ssh23:15:121
1156299394,24cyclictest4304-21ssh23:55:150
1156299394,24cyclictest4304-21ssh23:55:150
1159199383,22cyclictest28366irq/58-eth000:00:153
11583993818,14cyclictest11081-21ssh23:05:122
1156299382,28cyclictest30839-21ssh22:40:130
11591993731,3cyclictest28366irq/58-eth000:35:153
1158399372,18cyclictest17636-21grep19:15:202
11562993711,19cyclictest12-21ksoftirqd/020:05:120
1159199362,22cyclictest28366irq/58-eth023:25:133
1159199362,12cyclictest0-21swapper/323:40:123
1158399362,27cyclictest23453-21cut19:20:202
1159199353,25cyclictest28366irq/58-eth021:50:123
1159199353,25cyclictest28366irq/58-eth021:50:123
1156299352,25cyclictest5612-21kthreadcore20:10:200
11562993511,15cyclictest633-21Async12:11:530
1159199343,19cyclictest28366irq/58-eth022:15:143
11591993426,5cyclictest28366irq/58-eth023:15:133
1158399342,6cyclictest0-21swapper/200:26:242
1156299342,24cyclictest16463-21kthreadcore20:55:200
1159199334,15cyclictest28366irq/58-eth023:35:133
1158399332,25cyclictest6289-21ssh00:13:222
1158399332,14cyclictest27921-21cut21:05:202
1157199331,25cyclictest17150-21ssh00:20:131
1159199321,17cyclictest28366irq/58-eth019:50:133
11571993220,9cyclictest422-21ssh23:35:131
1157199321,24cyclictest24930-21sh21:25:121
1156299322,22cyclictest26200-21ssh00:25:120
1156299322,22cyclictest1352-21kthreadcore19:30:200
1156299322,21cyclictest7537-21kthreadcore21:15:200
1157199319,17cyclictest21-21ksoftirqd/122:30:111
1158399306,8cyclictest27-21ksoftirqd/223:52:212
1158399303,19cyclictest0-21swapper/200:15:142
1158399303,13cyclictest22346irq/40-dwc2_hso23:10:132
1158399301,9cyclictest19919-21kthreadcore19:50:202
1158399301,20cyclictest2131-21ssh23:00:142
1158399301,11cyclictest5613-21kthreadcore20:10:192
11571993021,3cyclictest11976-21apt-get00:35:141
11571993019,7cyclictest27453-21apt-get20:00:131
11571993017,9cyclictest17652-21apt-get22:15:031
11562993023,3cyclictest12-21ksoftirqd/023:50:120
1156299302,22cyclictest10782-21ssh22:28:030
1156299302,21cyclictest27286-21latency23:30:220
1156299302,20cyclictest10382-21kthreadcore20:15:200
1156299302,20cyclictest10382-21kthreadcore20:15:200
11591992918,7cyclictest0-21swapper/322:00:143
1159199291,22cyclictest28366irq/58-eth019:40:123
1158399298,9cyclictest27-21ksoftirqd/223:20:102
1158399298,9cyclictest27-21ksoftirqd/223:20:102
1158399295,15cyclictest27-21ksoftirqd/222:55:152
1158399295,15cyclictest27-21ksoftirqd/222:55:142
1158399293,17cyclictest22346irq/40-dwc2_hso22:20:112
1158399292,19cyclictest28474-21cut20:00:202
11583992917,8cyclictest0-21swapper/220:30:122
1158399291,21cyclictest1862-21kthreadcore20:05:202
1157199296,14cyclictest11093-21ssh23:05:131
11571992919,7cyclictest12953-21apt-get19:45:121
1156299292,7cyclictest131rcu_preempt23:13:220
1156299292,19cyclictest19920-21kthreadcore19:50:200
1156299292,19cyclictest14491-21grep19:10:200
11562992919,5cyclictest29734-21diskmemload00:35:150
1159199283,16cyclictest0-21swapper/300:20:223
1159199282,19cyclictest0-21swapper/322:27:203
1159199282,16cyclictest28366irq/58-eth023:00:123
1158399286,13cyclictest27-21ksoftirqd/221:20:472
1158399286,12cyclictest131rcu_preempt21:10:212
1158399285,9cyclictest27-21ksoftirqd/220:15:152
1158399285,9cyclictest27-21ksoftirqd/220:15:152
1158399284,3cyclictest131rcu_preempt00:00:022
1158399284,3cyclictest131rcu_preempt00:00:022
1158399283,11cyclictest131rcu_preempt22:07:332
1158399282,8cyclictest27-21ksoftirqd/221:25:082
1158399281,21cyclictest23608-21ssh00:22:242
1158399281,20cyclictest13601-21kthreadcore19:10:202
1157199284,15cyclictest14038-21sed00:00:191
1157199282,20cyclictest14786-21ssh23:25:131
1157199282,19cyclictest23454-21tr19:20:201
1156299285,11cyclictest11161-21kworker/u8:022:45:140
1156299282,19cyclictest11463-21kthreadcore00:15:210
11562992816,4cyclictest18581-21ssh23:45:130
11562992815,4cyclictest12-21ksoftirqd/022:55:130
11562992815,4cyclictest12-21ksoftirqd/022:55:130
11562992812,7cyclictest12-21ksoftirqd/020:00:130
1159199276,6cyclictest28366irq/58-eth021:40:203
1159199273,15cyclictest28366irq/58-eth000:25:123
1159199273,15cyclictest0-21swapper/319:15:203
1159199272,18cyclictest0-21swapper/322:43:213
1159199272,18cyclictest0-21swapper/300:10:143
1159199271,9cyclictest0-21swapper/323:55:213
1159199271,9cyclictest0-21swapper/323:55:213
1158399276,7cyclictest27-21ksoftirqd/200:38:132
1158399276,12cyclictest131rcu_preempt21:55:302
1158399275,11cyclictest131rcu_preempt23:45:212
1158399272,9cyclictest6762-21ssh21:15:142
11571992719,3cyclictest10641-21apt-get21:35:131
11571992718,4cyclictest8133-21apt-get19:40:121
1156299278,8cyclictest12-21ksoftirqd/021:55:130
1156299273,15cyclictest18126-21rm22:15:130
11562992715,4cyclictest0-21swapper/019:45:130
1156299271,20cyclictest12865-21ssh22:30:130
1159199268,3cyclictest0-21swapper/323:47:203
1159199263,19cyclictest28366irq/58-eth021:35:143
1159199263,15cyclictest0-21swapper/320:00:203
1159199263,15cyclictest0-21swapper/319:20:203
1159199263,14cyclictest0-21swapper/321:55:143
1159199263,14cyclictest0-21swapper/321:05:203
1159199262,6cyclictest0-21swapper/322:57:203
1159199262,6cyclictest0-21swapper/322:57:203
1159199261,9cyclictest28366irq/58-eth020:30:123
1159199261,7cyclictest0-21swapper/321:10:253
1159199261,11cyclictest0-21swapper/322:30:213
1158399268,3cyclictest131rcu_preempt19:35:342
1158399265,13cyclictest27-21ksoftirqd/222:37:012
1158399263,9cyclictest27-21ksoftirqd/221:54:192
1158399263,9cyclictest27-21ksoftirqd/221:54:192
1158399263,17cyclictest22346irq/40-dwc2_hso23:15:122
11583992610,6cyclictest0-21swapper/222:47:482
11571992621,2cyclictest15544-21apt-get20:55:131
11571992619,3cyclictest18570-21apt-get23:45:131
11571992615,8cyclictest32424-21ssh22:05:131
1156299262,18cyclictest29734-21diskmemload21:26:200
1156299262,18cyclictest18759-21ssh21:39:200
11562992614,8cyclictest134-21systemd-journal00:21:390
1159199254,17cyclictest28366irq/58-eth021:20:133
1159199252,8cyclictest0-21swapper/322:05:143
1159199252,5cyclictest0-21swapper/323:20:213
1159199252,5cyclictest0-21swapper/323:20:213
1159199252,14cyclictest28366irq/58-eth023:30:123
1158399257,7cyclictest27-21ksoftirqd/222:43:492
1158399257,11cyclictest27-21ksoftirqd/221:47:342
1158399256,9cyclictest27-21ksoftirqd/221:32:372
1158399256,11cyclictest27-21ksoftirqd/222:30:442
1158399255,8cyclictest27-21ksoftirqd/200:01:122
1158399255,3cyclictest131rcu_preempt22:10:232
1158399253,9cyclictest27-21ksoftirqd/222:51:482
1158399252,4cyclictest27-21ksoftirqd/221:00:192
1158399251,18cyclictest0-21swapper/220:25:122
1158399251,15cyclictest27-21ksoftirqd/223:25:422
1157199258,9cyclictest0-21swapper/123:03:231
11571992515,7cyclictest7092-21if_eth021:15:181
1156299251,17cyclictest28788-21ssh21:45:130
1159199246,8cyclictest0-21swapper/320:45:133
1159199243,8cyclictest0-21swapper/321:33:203
1159199243,15cyclictest0-21swapper/320:05:203
1159199243,14cyclictest28366irq/58-eth019:30:123
1158399246,10cyclictest27-21ksoftirqd/200:31:472
1158399244,7cyclictest27-21ksoftirqd/221:41:012
1158399242,16cyclictest0-21swapper/220:20:202
11583992418,4cyclictest11070-21ssh23:40:202
1158399240,2cyclictest131rcu_preempt00:05:232
1158399240,15cyclictest23332-21ssh22:00:132
1157199245,5cyclictest11632-21kthreadcore22:10:221
1157199244,4cyclictest21-21ksoftirqd/122:40:121
1157199244,4cyclictest21-21ksoftirqd/121:30:231
11571992417,3cyclictest15853-21apt-get21:20:131
11571992412,9cyclictest10075-21ssh23:40:171
1156299242,5cyclictest131rcu_preempt22:10:130
1159199236,5cyclictest28366irq/58-eth023:10:183
1159199233,13cyclictest0-21swapper/322:20:133
1159199231,11cyclictest28366irq/58-eth019:55:143
11591992310,3cyclictest0-21swapper/322:50:103
1158399236,11cyclictest27-21ksoftirqd/219:55:332
1158399233,2cyclictest131rcu_preempt19:45:162
1158399232,15cyclictest0-21swapper/219:40:202
1158399231,9cyclictest0-21swapper/221:36:062
1158399231,4cyclictest271ksoftirqd/220:35:202
1158399231,12cyclictest0-21swapper/222:17:252
1158399230,15cyclictest5907-21find20:45:122
1157199237,3cyclictest0-21swapper/122:36:441
1157199235,5cyclictest21-21ksoftirqd/100:05:031
1157199234,4cyclictest21-21ksoftirqd/123:55:141
1157199234,4cyclictest21-21ksoftirqd/123:55:141
1157199234,4cyclictest21-21ksoftirqd/100:28:091
1157199233,4cyclictest21-21ksoftirqd/122:45:461
1157199233,3cyclictest21-21ksoftirqd/123:11:081
1157199232,4cyclictest21-21ksoftirqd/121:52:021
1157199232,4cyclictest21-21ksoftirqd/121:52:021
11571992318,3cyclictest32237-21apt-get20:05:121
11571992314,7cyclictest9268-21memory23:20:251
11571992314,7cyclictest9268-21memory23:20:251
11571992312,7cyclictest9431-21apt-get20:15:131
11571992312,7cyclictest9431-21apt-get20:15:131
1157199231,18cyclictest8056-21sh00:15:141
11571992310,9cyclictest25157-21apt-get21:05:131
1157199230,8cyclictest31130-21cat21:10:201
1159199226,6cyclictest0-21swapper/322:10:383
1159199221,18cyclictest28366irq/58-eth022:35:113
1158399221,15cyclictest0-21swapper/220:50:032
1158399221,12cyclictest0-21swapper/222:27:492
1158399221,11cyclictest131rcu_preempt20:55:272
1157199224,4cyclictest25761-21tune2fs22:55:171
1157199224,4cyclictest25761-21tune2fs22:55:171
1157199223,7cyclictest4204-21kthreadcore19:35:171
1157199223,4cyclictest21-21ksoftirqd/122:51:431
1157199223,4cyclictest21-21ksoftirqd/122:26:451
1157199223,4cyclictest21-21ksoftirqd/100:32:121
11571992212,8cyclictest27696-21ssh23:50:121
1157199221,13cyclictest17767-21sed20:20:231
1157199220,9cyclictest20402-21cpu21:00:131
1157199220,8cyclictest20460-21cat19:15:301
1156299229,10cyclictest32280-21wc20:35:260
1156299222,8cyclictest31446-21ssh00:10:140
1156299222,16cyclictest30148-21ssh21:10:140
11562992216,3cyclictest121ksoftirqd/023:40:120
1156299221,18cyclictest25736-21ls22:35:280
1156299221,16cyclictest28393-21latency_hist21:45:010
1159199213,16cyclictest0-21swapper/319:25:203
1157199213,4cyclictest21-21ksoftirqd/121:57:591
1157199213,4cyclictest21-21ksoftirqd/119:10:021
1157199212,4cyclictest21-21ksoftirqd/122:00:461
11571992115,2cyclictest4648-21apt-get20:10:141
11571992115,2cyclictest26144-21apt-get19:25:131
11571992114,3cyclictest10752-21apt-get20:50:121
11571992111,7cyclictest32388-21nfs4_client21:45:241
1156299217,5cyclictest131rcu_preempt22:20:180
1156299216,3cyclictest131rcu_preempt00:30:270
11562992114,4cyclictest2822-21cut20:40:210
1156299211,14cyclictest5335-21ssh21:50:120
1156299211,14cyclictest5335-21ssh21:50:110
1159199206,5cyclictest0-21swapper/319:45:133
1159199203,10cyclictest0-21swapper/320:10:203
1159199202,5cyclictest0-21swapper/320:15:203
1159199202,5cyclictest0-21swapper/320:15:203
11591992011,5cyclictest0-21swapper/300:30:143
11591992010,7cyclictest0-21swapper/323:50:183
1158399201,9cyclictest1252-21fschecks_count20:40:172
1157199203,4cyclictest21-21ksoftirqd/120:43:201
11571992010,5cyclictest30948-21apt-get19:30:131
1156299204,10cyclictest0-21swapper/019:55:150
1156299202,13cyclictest16484-21ssh22:50:140
1156299201,5cyclictest0-21swapper/023:15:040
11562992012,6cyclictest16907-21memory00:00:240
11562992010,7cyclictest24656-21memory19:20:210
1159199199,8cyclictest0-21swapper/322:46:263
1159199193,9cyclictest0-21swapper/320:55:203
1159199192,11cyclictest0-21swapper/320:40:203
1159199192,10cyclictest28366irq/58-eth021:25:133
11591991910,6cyclictest0-21swapper/300:09:263
1157199196,4cyclictest0-21swapper/119:10:111
1157199195,10cyclictest5916-21apt-get20:45:131
1157199194,4cyclictest21-21ksoftirqd/119:55:211
1157199192,11cyclictest0-21swapper/120:30:131
1156299199,7cyclictest12410-21cut23:05:200
1156299196,3cyclictest131rcu_preempt19:20:010
1156299192,11cyclictest0-21swapper/021:05:130
1156299192,10cyclictest0-21swapper/023:38:220
1156299191,3cyclictest0-21swapper/022:07:330
11562991910,7cyclictest12-21ksoftirqd/023:25:120
11562991910,6cyclictest0-21swapper/020:20:220
11562991910,5cyclictest9584-21mailstats23:20:300
11562991910,5cyclictest9584-21mailstats23:20:300
1159199183,9cyclictest0-21swapper/320:50:203
1159199182,11cyclictest0-21swapper/320:35:203
1159199182,10cyclictest0-21swapper/321:00:133
1159199182,10cyclictest0-21swapper/320:20:043
1158399181,8cyclictest11504-21latency_hist19:10:022
1157199186,9cyclictest0-21swapper/119:50:121
11571991812,4cyclictest0-21swapper/120:35:131
1156299184,10cyclictest24934-21awk21:05:010
11562991811,4cyclictest19337-21tune2fs20:25:150
1156299181,12cyclictest26875-21ssh22:00:240
1159199172,10cyclictest0-21swapper/319:35:133
1159199171,10cyclictest0-21swapper/320:25:143
1157199179,6cyclictest22439-21sed20:25:211
1156299171,5cyclictest0-21swapper/019:25:200
1156299171,12cyclictest8463-21irqcore19:40:170
1156299171,11cyclictest7912-21sendmail-msp19:40:010
11562991710,5cyclictest6191-21if_eth020:45:170
11562991610,4cyclictest20143-1kworker/0:0H20:30:150
1156299151,10cyclictest11655-21wc19:10:020
1159199111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional