You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue May 12, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13737996847,10cyclictest35966irq/58-eth000:00:123
1371099617,37cyclictest19411-21ssh23:45:140
1372799551,47cyclictest0-21swapper/223:05:122
1372799550,51cyclictest10349-21apt-get19:40:132
1372799542,46cyclictest0-21swapper/221:25:132
1372799523,43cyclictest3385-21apt-get23:00:142
1372799523,43cyclictest3385-21apt-get23:00:132
1372799522,34cyclictest0-21swapper/223:52:352
1371499513,44cyclictest10398-21ssh22:10:141
1371499511,38cyclictest6965-21ssh21:50:111
1371099502,36cyclictest12303-21ssh23:05:130
1372799491,44cyclictest0-21swapper/221:40:142
1372799490,44cyclictest1464-21apt-get23:35:122
1371499492,4cyclictest131rcu_preempt22:20:111
1371099485,32cyclictest19377-21ssh22:15:130
1372799462,40cyclictest27346irq/40-dwc2_hso00:35:142
1372799461,40cyclictest2835-21apt-get00:30:142
1371499454,33cyclictest28309-21ssh23:50:151
1371499451,35cyclictest32310-21ssh21:10:151
1372799442,36cyclictest22973-21apt-get22:35:152
1371499443,34cyclictest13987-21ssh22:30:141
1371099441,35cyclictest24970-21ssh22:00:150
1371499431,31cyclictest26847-21ssh22:55:131
1373799416,24cyclictest35966irq/58-eth022:10:143
1372799419,28cyclictest29835-21needreboot00:25:242
13727994131,7cyclictest27346irq/40-dwc2_hso22:20:102
1372799412,33cyclictest27346irq/40-dwc2_hso21:45:152
1372799411,37cyclictest32301-21ssh21:10:132
1372799411,36cyclictest0-21swapper/200:00:122
1372799411,35cyclictest0-21swapper/223:45:152
1372799411,31cyclictest0-21swapper/221:15:142
1372799411,31cyclictest0-21swapper/220:35:132
1372799410,37cyclictest0-21swapper/200:05:152
1372799410,36cyclictest0-21swapper/222:30:122
1372799400,36cyclictest26073-21apt-get20:30:112
1372799400,24cyclictest31886-21diskmemload23:33:152
1371499405,26cyclictest13707-21ssh00:00:121
1372799391,34cyclictest0-21swapper/219:15:112
1372799391,28cyclictest1386-21apt-get22:05:122
1372799391,24cyclictest0-21swapper/221:31:222
1372799390,35cyclictest30213-21apt-get23:15:142
1372799390,28cyclictest31886-21diskmemload23:57:482
1372799390,26cyclictest7964-21cut19:35:212
13714993920,7cyclictest21-21ksoftirqd/123:10:131
1371099392,28cyclictest6896-21ssh23:20:110
1373799381,22cyclictest35966irq/58-eth021:10:133
1372799381,34cyclictest20092-21fschecks_count19:50:172
1372799381,33cyclictest32013-21apt-get22:40:142
1372799381,32cyclictest13927-21apt-get19:10:122
1372799381,30cyclictest17166-21ssh00:20:132
1372799380,30cyclictest0-21swapper/220:15:142
1372799380,23cyclictest31886-21diskmemload23:43:192
1372799380,23cyclictest0-21swapper/222:56:582
1371099383,24cyclictest31886-21diskmemload23:25:130
1373799375,27cyclictest35966irq/58-eth023:20:133
1372799371,33cyclictest12335-21apt-get21:35:152
1372799371,32cyclictest29692-21df20:00:142
1372799371,25cyclictest0-21swapper/222:15:132
1372799370,34cyclictest27422-21grep19:20:272
1372799370,33cyclictest13709-21cyclictest20:40:132
1372799370,33cyclictest13709-21cyclictest20:40:122
1372799370,33cyclictest0-21swapper/221:05:132
1371099373,21cyclictest753-21sendmail-mta20:25:130
13737993615,11cyclictest35966irq/58-eth023:00:143
13737993615,11cyclictest35966irq/58-eth023:00:143
1372799361,31cyclictest8500-21memory22:25:262
1372799361,30cyclictest0-21swapper/220:45:122
1372799360,32cyclictest21261-21/usr/sbin/munin20:25:132
1372799360,31cyclictest17826-21apt-get22:50:162
1371499362,4cyclictest0-21swapper/120:10:141
1373799354,26cyclictest35966irq/58-eth021:45:143
1372799352,30cyclictest8140-21apt-get00:15:122
1372799351,32cyclictest12324-21/usr/sbin/munin22:45:242
1372799350,32cyclictest12947-21df_inode20:50:162
1372799350,26cyclictest10652-21proc_pri21:50:262
1373799342,21cyclictest35966irq/58-eth022:40:133
1372799341,30cyclictest729-21/usr/sbin/munin19:30:242
1372799341,30cyclictest729-21/usr/sbin/munin19:30:242
1372799341,30cyclictest32096-21cut00:10:172
1372799341,30cyclictest0-21swapper/223:10:122
1372799341,29cyclictest24681-21latency_hist22:00:022
1372799341,25cyclictest5485-21processes20:05:272
1372799340,31cyclictest24404-21ssh21:22:282
1372799340,31cyclictest18117-21ssh22:13:592
1372799340,19cyclictest0-21swapper/223:29:322
1371499341,27cyclictest24973-21ssh23:30:131
1372799331,24cyclictest25874-21/usr/sbin/munin21:00:222
1372799330,29cyclictest15172-21munin-node19:45:122
1372799330,29cyclictest0-21swapper/220:55:132
1371499331,20cyclictest1493-21ssh23:35:131
1373799324,16cyclictest0-21swapper/321:50:113
1373799321,16cyclictest35966irq/58-eth000:15:133
1372799321,28cyclictest30258-21ssh22:01:092
1372799321,27cyclictest0-21swapper/219:25:132
1372799320,25cyclictest15010-21ssh23:24:202
1371499322,26cyclictest1389-21ssh22:05:121
1373799311,18cyclictest35966irq/58-eth023:10:133
1372799311,27cyclictest0-21swapper/219:55:132
13714993114,12cyclictest13709-21cyclictest00:20:131
1371099319,15cyclictest12-21ksoftirqd/023:55:140
1373799305,13cyclictest0-21swapper/300:35:133
1373799302,16cyclictest0-21swapper/300:05:133
13737993017,7cyclictest35966irq/58-eth022:20:123
13737993015,11cyclictest35966irq/58-eth023:50:153
1372799301,20cyclictest19964-21grep20:20:262
1371499296,12cyclictest21-21ksoftirqd/123:55:141
13714992914,11cyclictest23052-21ssh22:35:151
1371099296,12cyclictest630-21Async09:36:310
1373799286,12cyclictest0-21swapper/322:55:123
1373799283,16cyclictest0-21swapper/300:25:163
13737992820,5cyclictest35966irq/58-eth023:25:143
13737992812,11cyclictest35966irq/58-eth023:35:123
1372799280,24cyclictest10176-21/usr/sbin/munin20:10:252
1371499281,24cyclictest26259-21ssh00:25:151
13737992713,9cyclictest35966irq/58-eth023:30:133
1371499276,13cyclictest0-21swapper/120:20:141
1371499273,17cyclictest844-21munin-node21:30:131
1371499272,19cyclictest4947-21ssh23:18:161
1371099273,6cyclictest12-21ksoftirqd/023:40:270
1371099272,19cyclictest6066-21ssh23:19:360
1371099272,19cyclictest16167-21sh22:48:350
1371099271,4cyclictest12-21ksoftirqd/000:04:530
1371099271,20cyclictest22681-21ssh00:05:150
1373799263,16cyclictest0-21swapper/300:23:363
1373799261,17cyclictest0-21swapper/322:25:143
1371499263,19cyclictest21-21ksoftirqd/120:40:131
1371499263,19cyclictest21-21ksoftirqd/120:40:131
1371499262,19cyclictest7082-21ssh22:43:361
1371099269,8cyclictest12-21ksoftirqd/022:54:160
1371099262,19cyclictest29623-21ssh22:37:350
1371099262,18cyclictest17191-21ssh00:20:150
1371099262,17cyclictest15939-21ssh21:55:140
1371099262,15cyclictest32025-21ssh22:40:130
1371099261,19cyclictest26830-21apt-get22:55:120
1373799252,17cyclictest0-21swapper/323:58:363
13737992513,9cyclictest35966irq/58-eth021:15:143
13714992511,10cyclictest21-21ksoftirqd/121:20:141
1371099253,5cyclictest12-21ksoftirqd/023:00:510
1371099253,5cyclictest12-21ksoftirqd/023:00:510
1371099251,17cyclictest21280-21ssh23:10:130
1373799245,10cyclictest35966irq/58-eth020:10:133
1373799243,17cyclictest35966irq/58-eth021:25:133
1373799243,15cyclictest35966irq/58-eth022:15:123
1371499246,9cyclictest0-21swapper/119:25:131
1371499242,11cyclictest21-21ksoftirqd/119:20:121
13714992410,3cyclictest131rcu_preempt00:05:191
1371099245,3cyclictest131rcu_preempt21:52:240
1371099245,3cyclictest131rcu_preempt20:30:260
1371099243,7cyclictest12-21ksoftirqd/021:48:070
1371099243,6cyclictest22793-21ssh21:20:330
1371099243,5cyclictest12-21ksoftirqd/021:11:520
1371099242,16cyclictest12588-21ssh21:15:280
13710992412,8cyclictest6778-21apt-get20:10:120
13710992412,7cyclictest0-21swapper/021:39:450
13710992410,9cyclictest13063-21ssh00:15:370
1373799234,12cyclictest36066irq/59-eth022:35:123
1373799233,15cyclictest35966irq/58-eth000:10:123
1373799232,3cyclictest0-21swapper/300:30:133
13714992312,8cyclictest3886-21ssh00:10:311
1371099238,3cyclictest131rcu_preempt23:35:080
1371099237,8cyclictest151rcuc/022:10:440
1371099237,6cyclictest12-21ksoftirqd/022:22:440
1371099237,11cyclictest151rcuc/021:42:430
1371099235,4cyclictest12-21ksoftirqd/022:05:590
1371099234,8cyclictest12-21ksoftirqd/023:52:420
1371099233,5cyclictest12-21ksoftirqd/023:31:550
1371099233,5cyclictest12-21ksoftirqd/021:25:050
1371099232,4cyclictest12-21ksoftirqd/022:32:410
1371099232,15cyclictest6355-21ssh00:13:210
13710992310,4cyclictest12-21ksoftirqd/000:29:400
1373799223,13cyclictest0-21swapper/323:40:363
1373799223,13cyclictest0-21swapper/322:00:153
1373799221,11cyclictest0-21swapper/322:55:023
1373799221,11cyclictest0-21swapper/320:20:133
13737992210,7cyclictest35966irq/58-eth023:15:133
1371499228,3cyclictest131rcu_preempt21:25:321
1371499223,14cyclictest30426-21ssh21:45:131
1371499221,17cyclictest8701-21ssh22:25:291
1371499221,16cyclictest2866-21ssh00:30:151
13714992210,9cyclictest31886-21diskmemload22:50:261
1371499220,19cyclictest12108-21ls00:35:171
1371099226,5cyclictest12-21ksoftirqd/019:45:250
1371099226,3cyclictest131rcu_preempt20:40:180
1371099226,3cyclictest131rcu_preempt20:40:170
1371099225,9cyclictest12-21ksoftirqd/020:20:330
1371099225,7cyclictest131rcu_preempt22:27:470
1371099225,3cyclictest131rcu_preempt00:30:150
1371099221,17cyclictest17807-21cut20:55:170
1371099221,16cyclictest0-21swapper/019:30:120
1371099221,16cyclictest0-21swapper/019:30:110
1373799219,6cyclictest0-21swapper/321:20:133
1373799213,15cyclictest0-21swapper/322:48:363
1373799212,16cyclictest35966irq/58-eth021:30:123
1373799212,14cyclictest35966irq/58-eth019:30:133
1373799212,14cyclictest35966irq/58-eth019:30:133
1373799212,14cyclictest0-21swapper/322:30:123
1371499212,15cyclictest1970-21apt-get20:05:111
13714992113,6cyclictest16621-21ssh21:35:311
13714992112,6cyclictest8277-21ssh23:00:341
13714992112,6cyclictest8277-21ssh23:00:331
1371499211,17cyclictest15102-21ls19:45:021
1371499211,15cyclictest19610-21cut23:45:161
1371499211,11cyclictest710-21cron23:45:011
1371499210,17cyclictest14949-21ls20:15:231
1371099215,3cyclictest131rcu_preempt21:05:230
1371099213,3cyclictest12-21ksoftirqd/020:35:140
1371099213,2cyclictest131rcu_preempt20:20:010
1371099211,5cyclictest12-21ksoftirqd/000:35:130
1371099210,6cyclictest5436-21ntp_states20:05:250
1373799208,8cyclictest0-21swapper/323:05:273
1373799202,11cyclictest0-21swapper/319:15:123
1373799201,9cyclictest35966irq/58-eth020:25:123
1373799201,15cyclictest0-21swapper/323:45:123
13737992010,7cyclictest35966irq/58-eth021:55:133
1371499209,8cyclictest21525-21df_inode21:40:171
1371499208,10cyclictest21-21ksoftirqd/123:29:371
1371499202,15cyclictest21-21ksoftirqd/123:20:111
13714992012,6cyclictest0-21swapper/123:05:161
1371499201,16cyclictest32083-21kthreadcore20:35:201
1371499201,13cyclictest19965-21apt-get19:50:131
1371499201,11cyclictest0-21swapper/122:19:021
13714992010,7cyclictest26236-21cut22:00:221
1371099204,3cyclictest131rcu_preempt19:35:210
1371099202,5cyclictest12-21ksoftirqd/020:45:160
1371099202,4cyclictest12-21ksoftirqd/019:15:140
1371099201,6cyclictest12-21ksoftirqd/020:50:120
13710992012,6cyclictest23461-21ntp_states19:50:260
1371099201,15cyclictest23590-21apt-get19:20:120
1371099200,7cyclictest12-21ksoftirqd/019:10:160
1373799192,9cyclictest0-21swapper/319:50:143
1373799191,12cyclictest0-21swapper/320:35:023
1371499196,4cyclictest131rcu_preempt00:15:211
1371499191,3cyclictest0-21swapper/119:15:111
13714991910,6cyclictest9406-21kthreadcore22:45:191
1371099195,2cyclictest131rcu_preempt20:05:000
1371099191,4cyclictest12-21ksoftirqd/019:25:180
1373799182,6cyclictest0-21swapper/321:45:013
1373799182,10cyclictest0-21swapper/321:00:143
1373799182,10cyclictest0-21swapper/319:35:143
1373799182,10cyclictest0-21swapper/319:25:133
1371499189,5cyclictest15941-21cpu21:55:131
1371499184,11cyclictest29883-21kthreadcore21:05:211
1371499182,13cyclictest9614-21kthreadcore21:15:211
1371499181,5cyclictest13192-21grep20:50:191
13710991812,4cyclictest0-21swapper/019:40:270
1371099181,14cyclictest25041-21ls19:55:180
1371099180,5cyclictest12-21ksoftirqd/021:00:160
1373799178,6cyclictest35966irq/58-eth021:35:133
1373799178,5cyclictest0-21swapper/322:05:123
1373799173,12cyclictest0-21swapper/320:15:143
1373799173,10cyclictest35966irq/58-eth020:40:123
1373799173,10cyclictest35966irq/58-eth020:40:113
1373799172,10cyclictest0-21swapper/321:05:153
1373799172,10cyclictest0-21swapper/320:50:153
1373799172,10cyclictest0-21swapper/320:45:143
1373799172,10cyclictest0-21swapper/319:55:153
1373799172,10cyclictest0-21swapper/319:20:133
1373799171,9cyclictest0-21swapper/320:05:143
1371499179,5cyclictest22507-21apt-get21:00:121
1371499175,9cyclictest29600-21sort20:00:121
1371499171,8cyclictest5527-21/usr/sbin/munin19:35:251
1373799162,10cyclictest0-21swapper/320:55:143
1371499165,8cyclictest29719-21timerandwakeup20:30:281
1371499163,11cyclictest27762-21kthreadcore19:55:211
1371499161,12cyclictest710-21cron20:50:011
1371499161,12cyclictest17805-21fschecks_count20:55:161
1371499161,12cyclictest17335-21ntp_192.168.11419:10:241
1371499161,11cyclictest16848-21kthreadcore19:45:201
1371499161,11cyclictest13803-21latency_hist19:10:011
1371499161,11cyclictest13803-21latency_hist19:10:001
1373799152,7cyclictest0-21swapper/319:45:143
1371499154,9cyclictest22811-21kthreadcore20:25:191
1371499151,12cyclictest747-21apt-get19:30:111
1371499151,12cyclictest747-21apt-get19:30:111
1373799141,6cyclictest0-21swapper/320:00:283
1372799141,9cyclictest13851-21latency_hist19:10:022
1372799141,9cyclictest13851-21latency_hist19:10:022
1371099141,9cyclictest13796-21latency_hist19:10:010
1371099141,9cyclictest13796-21latency_hist19:10:000
1373799131,8cyclictest0-21swapper/320:35:123
1373799122,6cyclictest0-21swapper/319:40:143
1373799122,6cyclictest0-21swapper/319:10:143
1373799111,7cyclictest0-21swapper/319:10:013
1373799111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional