You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 22:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Thu Jan 01, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1496199626,11cyclictest131rcu_preempt07:55:102
1493899612,47cyclictest30054-21ssh10:20:170
1493899582,9cyclictest131rcu_preempt09:25:120
1493899562,12cyclictest8643-21ssh12:15:130
1494999524,29cyclictest24541-21ssh10:35:121
1493899523,35cyclictest27917-21ssh10:55:120
14949995037,5cyclictest16817-21rm11:25:141
14961994917,17cyclictest27-21ksoftirqd/210:55:142
1493899492,38cyclictest5303-21sh09:30:150
14961994421,15cyclictest131rcu_preempt10:20:162
1494999441,33cyclictest4493-21ssh11:00:141
1497299433,33cyclictest28366irq/58-eth009:45:113
1494999431,35cyclictest26593-21ssh12:25:141
1496199412,32cyclictest21637-21ssh12:03:242
1493899412,4cyclictest131rcu_preempt11:00:150
1493899412,33cyclictest11166-21ssh11:40:140
1493899411,31cyclictest19777-21ssh09:20:130
14961994016,7cyclictest27-21ksoftirqd/212:15:132
14961994014,17cyclictest14925-21cyclictest12:35:152
1496199394,27cyclictest27-21ksoftirqd/209:50:142
1496199393,26cyclictest27-21ksoftirqd/209:55:122
1496199392,29cyclictest21263-21ntp_192.168.11412:20:262
1494999391,31cyclictest1159-21ssh10:40:131
1494999384,26cyclictest14409-21ssh12:00:141
1494999381,30cyclictest27932-21ssh10:55:131
1497299374,22cyclictest28366irq/58-eth011:30:123
1497299370,2cyclictest0-21swapper/312:20:133
1496199371,26cyclictest5306-21ssh09:30:142
1493899371,30cyclictest25749-21ssh11:30:120
1497299352,24cyclictest28366irq/58-eth010:20:173
1496199352,19cyclictest2275-21sh11:35:132
1496199351,25cyclictest23266-21ssh12:05:132
14972993420,7cyclictest0-21swapper/311:55:133
1497299341,3cyclictest0-21swapper/308:30:133
1496199343,17cyclictest22346irq/40-dwc2_hso10:40:132
1494999343,26cyclictest1096-21diskmemload12:10:141
1494999343,26cyclictest1096-21diskmemload12:10:141
1494999342,18cyclictest20116-21ssh11:45:141
1493899345,20cyclictest12-21ksoftirqd/011:20:130
1493899345,20cyclictest12-21ksoftirqd/011:20:130
1497299335,14cyclictest28366irq/58-eth012:00:143
1497299331,2cyclictest0-21swapper/307:55:093
14961993316,12cyclictest0-21swapper/207:20:132
14961993316,12cyclictest0-21swapper/207:20:122
1494999336,17cyclictest8723-21ssh09:50:141
1493899332,27cyclictest3085-21ssh12:30:140
1493899332,22cyclictest17526-21ssh12:36:240
1496199322,15cyclictest1096-21diskmemload12:34:252
1497299311,2cyclictest0-21swapper/309:30:153
1497299311,2cyclictest0-21swapper/308:10:113
1497299311,26cyclictest0-21swapper/308:55:143
1497299311,25cyclictest0-21swapper/311:15:133
1494999313,20cyclictest17655-21ssh09:55:131
1493899312,11cyclictest26889-21ntp_192.168.11412:05:250
14938993114,4cyclictest131rcu_preempt10:10:280
1497299306,16cyclictest28366irq/58-eth010:35:133
1497299302,2cyclictest0-21swapper/312:05:133
1497299301,6cyclictest0-21swapper/309:35:133
1497299301,2cyclictest0-21swapper/310:25:143
1497299301,2cyclictest0-21swapper/307:10:123
1497299301,24cyclictest0-21swapper/309:15:113
1497299295,21cyclictest28366irq/58-eth007:15:123
1497299292,23cyclictest0-21swapper/309:25:133
1497299292,16cyclictest28366irq/58-eth010:55:123
1497299291,2cyclictest0-21swapper/311:10:133
1497299291,21cyclictest0-21swapper/312:15:133
1497299290,2cyclictest0-21swapper/307:25:133
1496199295,15cyclictest22346irq/40-dwc2_hso07:30:172
1496199293,16cyclictest22346irq/40-dwc2_hso09:15:112
1496199291,20cyclictest22346irq/40-dwc2_hso11:20:142
1496199291,20cyclictest22346irq/40-dwc2_hso11:20:142
1496199291,15cyclictest23221-21ssh09:40:132
1494999296,14cyclictest1096-21diskmemload09:35:131
1494999292,19cyclictest15594-21ntp_192.168.11412:35:251
14938992917,7cyclictest0-21swapper/010:30:140
1493899291,20cyclictest2293-21ssh11:35:160
1497299283,19cyclictest0-21swapper/312:36:103
1497299283,17cyclictest0-21swapper/312:10:143
1497299283,17cyclictest0-21swapper/312:10:143
1497299281,2cyclictest0-21swapper/310:50:133
1497299280,2cyclictest0-21swapper/310:15:123
1497299280,2cyclictest0-21swapper/307:30:163
1496199284,14cyclictest12978-21apt08:15:132
1496199283,17cyclictest22346irq/40-dwc2_hso09:00:132
1496199283,17cyclictest22346irq/40-dwc2_hso08:05:112
1496199283,16cyclictest22346irq/40-dwc2_hso09:20:122
1496199282,20cyclictest0-21swapper/208:55:132
1497299272,17cyclictest28366irq/58-eth010:00:163
1497299272,17cyclictest28366irq/58-eth010:00:153
1497299271,8cyclictest0-21swapper/311:40:153
1497299271,3cyclictest0-21swapper/309:10:123
1497299271,2cyclictest0-21swapper/311:20:133
1497299271,2cyclictest0-21swapper/311:20:133
1497299271,2cyclictest0-21swapper/309:40:133
1497299271,21cyclictest0-21swapper/312:30:143
1497299270,2cyclictest0-21swapper/312:25:133
1497299270,2cyclictest0-21swapper/311:00:143
1497299270,2cyclictest0-21swapper/308:50:133
1496199273,15cyclictest22346irq/40-dwc2_hso08:40:122
1496199273,14cyclictest22259-21ssh11:10:132
1496199272,19cyclictest0-21swapper/211:25:142
1496199271,21cyclictest0-21swapper/208:25:142
1497299261,2cyclictest0-21swapper/310:45:133
1497299261,2cyclictest0-21swapper/310:05:113
1497299261,22cyclictest0-21swapper/309:21:523
1497299261,22cyclictest0-21swapper/309:00:323
1497299261,20cyclictest0-21swapper/311:50:173
1497299260,2cyclictest0-21swapper/310:10:123
1496199263,18cyclictest0-21swapper/212:26:252
1496199263,15cyclictest22346irq/40-dwc2_hso10:15:122
1496199263,15cyclictest22346irq/40-dwc2_hso07:15:122
1496199262,9cyclictest5354-21ssh12:11:252
1496199262,9cyclictest5354-21ssh12:11:252
14961992614,9cyclictest8812-21ssh09:13:242
1496199261,22cyclictest29003-21sh11:50:142
1497299251,6cyclictest0-21swapper/307:20:133
1497299251,6cyclictest0-21swapper/307:20:133
1497299251,4cyclictest0-21swapper/310:33:093
1497299251,2cyclictest0-21swapper/310:40:133
1497299251,21cyclictest0-21swapper/311:35:133
1497299251,1cyclictest0-21swapper/311:45:123
14961992512,10cyclictest748-21kthreadcore09:45:192
1496199251,17cyclictest22346irq/40-dwc2_hso11:15:122
14949992513,4cyclictest21-21ksoftirqd/111:35:191
1494999251,16cyclictest17652-21sh12:20:141
14938992519,3cyclictest531-21usb-storage09:15:110
1493899251,15cyclictest6570-21ssh10:25:130
1497299243,12cyclictest0-21swapper/307:35:143
1497299242,2cyclictest0-21swapper/308:00:123
1497299241,5cyclictest0-21swapper/309:55:083
1497299241,3cyclictest0-21swapper/307:45:143
1497299241,2cyclictest0-21swapper/311:05:283
1497299241,2cyclictest0-21swapper/309:50:393
1497299241,2cyclictest0-21swapper/308:15:123
1497299241,2cyclictest0-21swapper/307:50:123
1497299241,20cyclictest0-21swapper/311:25:303
1497299241,20cyclictest0-21swapper/308:35:133
1497299241,18cyclictest0-21swapper/308:40:123
1497299240,2cyclictest0-21swapper/309:05:113
1497299240,2cyclictest0-21swapper/308:25:133
1496199242,16cyclictest22346irq/40-dwc2_hso10:35:122
1496199242,14cyclictest0-21swapper/208:00:122
1496199241,10cyclictest11892-21ls07:40:152
1494999249,7cyclictest21-21ksoftirqd/110:31:251
1494999243,15cyclictest633-21Async12:11:531
1494999242,16cyclictest29051-21ssh11:50:161
14949992411,4cyclictest21-21ksoftirqd/111:15:161
1497299231,4cyclictest0-21swapper/308:20:283
1497299231,2cyclictest0-21swapper/308:45:273
1497299231,19cyclictest0-21swapper/308:05:273
1497299231,16cyclictest0-21swapper/307:40:123
1496199232,15cyclictest0-21swapper/207:50:112
14961992312,9cyclictest11830-21ssh11:03:372
14961992311,9cyclictest0-21swapper/208:45:242
1496199231,18cyclictest18340-21ls09:35:302
1496199231,14cyclictest21657-21kthreadcore11:45:212
1494999238,10cyclictest32110-21sh09:45:121
1494999235,9cyclictest12051-21ssh10:10:121
1494999234,4cyclictest131rcu_preempt10:06:281
1494999233,11cyclictest21-21ksoftirqd/110:04:351
1494999233,11cyclictest21-21ksoftirqd/110:04:351
1494999232,16cyclictest14295-21ssh09:15:231
1494999231,17cyclictest10131-21ssh10:45:151
14949992311,4cyclictest21-21ksoftirqd/112:15:171
1493899232,13cyclictest0-21swapper/007:15:120
1496199229,10cyclictest10191-21nfs_client10:25:252
1496199227,8cyclictest17553-21ps10:30:232
1496199223,12cyclictest261rcuc/210:45:132
1496199222,12cyclictest839-21munin-node08:20:122
1496199221,9cyclictest1335466ssh11:05:142
1496199221,5cyclictest20259-21timerwakeupswit07:45:282
14961992211,9cyclictest1137-21ssh10:03:042
14961992211,9cyclictest1137-21ssh10:03:042
1496199221,17cyclictest2956466ssh11:30:282
1496199220,9cyclictest8654-21kthreadcore11:55:232
1494999229,5cyclictest21-21ksoftirqd/108:50:321
1494999228,11cyclictest201rcuc/110:15:161
1494999223,4cyclictest131rcu_preempt07:45:121
14949992212,8cyclictest1387-21sshd11:34:251
14949992210,4cyclictest21-21ksoftirqd/108:00:271
1493899222,14cyclictest32172-21ssh12:26:250
1493899221,8cyclictest22556-21vmstat09:55:320
1496199219,10cyclictest32323-21latency_hist08:35:012
1496199217,12cyclictest1090-21cron09:10:012
1496199217,11cyclictest14414-21kthreadcore11:40:242
1496199213,15cyclictest22346irq/40-dwc2_hso09:25:132
14961992112,7cyclictest12017-21/usr/sbin/munin10:10:302
1494999219,3cyclictest21-21ksoftirqd/110:28:131
1494999219,2cyclictest131rcu_preempt12:31:261
1494999212,16cyclictest1096-21diskmemload11:55:131
14949992114,5cyclictest0-21swapper/110:50:151
14949992110,9cyclictest0-21swapper/109:10:251
1493899216,4cyclictest131rcu_preempt09:40:020
1493899216,3cyclictest131rcu_preempt10:35:220
1493899213,12cyclictest14925-21cyclictest10:15:120
14938992111,7cyclictest21081-21cron07:50:000
1493899211,16cyclictest31555-21ssh10:00:360
1493899211,16cyclictest31555-21ssh10:00:360
1493899211,10cyclictest0-21swapper/012:24:520
1496199202,9cyclictest17499-21kthreadcore08:50:242
14961992015,3cyclictest4711-21ssh10:05:202
14961992012,5cyclictest18866-21ssh10:50:042
1496199201,17cyclictest0-21swapper/207:25:292
14961992010,8cyclictest27-21ksoftirqd/207:35:182
1494999209,7cyclictest21-21ksoftirqd/108:15:121
1494999208,3cyclictest131rcu_preempt09:30:161
1494999208,3cyclictest131rcu_preempt07:10:251
1494999206,3cyclictest131rcu_preempt08:25:261
1494999204,4cyclictest131rcu_preempt09:42:441
1494999202,15cyclictest25964-21ssh11:10:271
1494999201,8cyclictest131rcu_preempt09:26:081
14949992012,6cyclictest28184-21nfs4_client07:20:231
14949992012,6cyclictest28184-21nfs4_client07:20:231
14949992012,6cyclictest22434-21cron08:25:011
1494999200,14cyclictest27786-21timerandwakeup09:00:321
1493899209,8cyclictest18244-21users11:05:320
1493899208,9cyclictest1-21systemd08:30:300
1493899201,8cyclictest26919-21ntp_states09:40:270
1493899201,7cyclictest25901-21/usr/sbin/munin11:10:260
1493899201,6cyclictest18834-21cat12:00:340
1493899201,3cyclictest0-21swapper/009:14:240
14938992011,7cyclictest5045-21fw_packets08:40:160
14938992011,6cyclictest171-21systemd-network11:45:100
14938992011,6cyclictest15242-21sshd10:45:500
14938992011,6cyclictest0-21swapper/009:45:290
14938992011,6cyclictest0-21swapper/008:00:290
1493899200,15cyclictest29129-21ls11:50:170
1496199192,13cyclictest22346irq/40-dwc2_hso07:10:122
1496199192,10cyclictest8629-21irqcore08:10:182
1496199191,9cyclictest436-21grep08:35:192
1494999198,3cyclictest53146usb-storage09:20:111
1494999197,4cyclictest21-21ksoftirqd/110:23:471
1494999193,3cyclictest131rcu_preempt11:22:491
1494999193,3cyclictest131rcu_preempt11:22:491
14949991914,3cyclictest0-21swapper/107:25:141
14949991913,4cyclictest0-21swapper/111:10:011
14949991911,5cyclictest27279-21timerwakeupswit12:05:291
1494999191,14cyclictest11182-21sh11:40:151
1494999191,13cyclictest23062-21timerwakeupswit08:55:291
1493899198,8cyclictest12524-21processes09:50:300
1493899193,13cyclictest2664-21kthreadcore07:30:220
1493899192,14cyclictest19166-21kthreadcore08:20:180
1493899191,3cyclictest0-21swapper/011:25:360
14938991911,5cyclictest5565-21usage_Anybus10:40:330
14938991911,5cyclictest338-21grep09:05:280
1493899191,15cyclictest21318-21apt-get07:50:110
1493899191,14cyclictest9082-21grep11:55:240
14938991910,6cyclictest0-21swapper/008:05:160
1496199181,8cyclictest15031-21latency_hist07:10:022
1494999185,4cyclictest131rcu_preempt09:05:171
1494999184,4cyclictest21-21ksoftirqd/108:45:211
1494999184,3cyclictest131rcu_preempt08:40:311
1494999182,10cyclictest839-21munin-node07:30:161
1494999181,8cyclictest21-21ksoftirqd/107:35:291
14949991813,3cyclictest0-21swapper/107:55:161
14949991812,4cyclictest0-21swapper/107:50:161
14949991812,4cyclictest0-21swapper/107:15:251
14949991811,4cyclictest12848-21cut07:40:191
14949991810,6cyclictest32322-21cat08:35:011
14949991810,5cyclictest7928-21cron08:10:011
1493899187,9cyclictest28669-21sed07:20:290
1493899187,9cyclictest28669-21sed07:20:290
1493899181,9cyclictest30261-21kthreadcore07:25:200
1493899181,4cyclictest12-21ksoftirqd/007:55:270
1493899181,14cyclictest23395-21kthreadcore08:25:200
1493899181,13cyclictest5573-21cut10:05:210
1493899181,12cyclictest11433-21sendmail-msp07:40:010
1497299171,4cyclictest0-21swapper/307:10:023
1493899178,7cyclictest19000-21apt-get10:50:150
1493899174,11cyclictest17338-21kthreadcore07:10:200
1493899171,8cyclictest24130-21grep09:00:140
1493899171,12cyclictest395-21irqstats12:10:210
1493899171,12cyclictest395-21irqstats12:10:200
1493899171,12cyclictest31152-21ssh11:15:120
1493899171,12cyclictest17954-21ntp_states08:50:270
1493899170,15cyclictest19424-21uniq08:55:160
1493899170,14cyclictest8238-21ls08:10:140
1493899170,13cyclictest13643-21kernelversion08:15:190
1494999162,9cyclictest0-21swapper/108:35:151
1493899161,12cyclictest14249-21munin-plugin-st08:50:010
1493899161,10cyclictest4710-21grep08:40:010
1494999151,10cyclictest14991-21latency_hist07:10:021
1493899151,11cyclictest14937-21latency_hist07:10:020
1493899151,11cyclictest11837-21fschecks_time07:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional