You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 05:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 17, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
986299664,37cyclictest4889-21ssh22:45:132
986299664,37cyclictest4889-21ssh22:45:132
9854996115,11cyclictest131rcu_preempt20:05:131
9854996010,13cyclictest131rcu_preempt21:45:111
9854996010,13cyclictest131rcu_preempt21:45:111
9862995523,21cyclictest27-21ksoftirqd/221:25:102
9845995519,3cyclictest0-21swapper/021:00:100
986299527,40cyclictest3292-21ssh23:20:132
984599526,32cyclictest5592-21ssh00:15:150
9854995130,11cyclictest131rcu_preempt20:20:121
9854995130,11cyclictest131rcu_preempt20:20:111
9874995024,17cyclictest35966irq/58-eth023:45:123
985499507,21cyclictest21-21ksoftirqd/123:45:121
985499502,7cyclictest131rcu_preempt19:55:121
986299495,34cyclictest19443-21ssh22:35:142
9854994922,20cyclictest21-21ksoftirqd/122:00:141
986299486,12cyclictest27346irq/40-dwc2_hso21:45:112
986299486,12cyclictest27346irq/40-dwc2_hso21:45:112
985499484,8cyclictest131rcu_preempt19:20:121
985499483,35cyclictest12326-21ssh21:55:131
9862994716,25cyclictest27346irq/40-dwc2_hso23:30:112
984599471,37cyclictest12367-21ssh23:25:120
9862994614,10cyclictest8631-21ssh21:35:122
985499463,33cyclictest32032-21sh23:00:131
986299452,37cyclictest27346irq/40-dwc2_hso23:25:142
984599455,7cyclictest53646usb-storage21:25:100
984599454,35cyclictest24931-21ssh22:20:130
9862994410,12cyclictest26587-21sh23:15:132
987499425,23cyclictest35966irq/58-eth000:20:133
984599427,27cyclictest15958-21ssh22:15:140
985499402,26cyclictest26561-21ssh23:15:121
987499394,23cyclictest35966irq/58-eth000:35:133
986299391,14cyclictest28357-21ssh22:40:132
9874993831,5cyclictest321rcuc/321:05:143
9874993814,16cyclictest35966irq/58-eth023:35:133
986299382,29cyclictest16581-21kthreadcore22:50:222
985499381,30cyclictest1476-21ssh22:25:141
987499372,19cyclictest35966irq/58-eth022:30:143
986299372,18cyclictest2502-21kthreadcore20:40:222
986299372,16cyclictest0-21swapper/223:46:222
986299371,16cyclictest28030-21diskmemload00:15:042
9845993722,8cyclictest26714-21apt-get21:45:120
9845993722,8cyclictest26714-21apt-get21:45:120
9874993613,15cyclictest321rcuc/321:00:103
986299362,16cyclictest0-21swapper/223:57:182
985499369,8cyclictest131rcu_preempt20:40:131
984599362,25cyclictest17682-21rm21:40:120
987499352,22cyclictest35966irq/58-eth022:15:143
987499352,18cyclictest35966irq/58-eth022:55:133
987499351,22cyclictest35966irq/58-eth023:25:143
986299351,13cyclictest19889-21ssh00:05:132
985499351,23cyclictest7135-21ssh23:40:131
986299347,10cyclictest0-21swapper/220:26:222
986299343,8cyclictest0-21swapper/219:10:222
984599343,24cyclictest8611-21ssh23:05:140
984599342,25cyclictest0-21swapper/020:15:130
987499333,17cyclictest35966irq/58-eth019:55:113
984599330,27cyclictest3324-21sh21:50:130
987499325,18cyclictest35966irq/58-eth000:05:133
987499321,22cyclictest0-21swapper/319:10:223
987499321,20cyclictest35966irq/58-eth022:00:133
9862993221,4cyclictest2746ksoftirqd/222:00:132
984599325,10cyclictest12-21ksoftirqd/023:55:140
984599323,21cyclictest8648-21diskmemload21:35:130
984599322,22cyclictest3374-21cut22:25:230
984599322,22cyclictest16580-21kthreadcore22:50:220
987499315,18cyclictest321rcuc/320:40:123
986299312,12cyclictest9887-21netstat19:40:222
986299312,12cyclictest9887-21netstat19:40:212
985499314,11cyclictest131rcu_preempt20:00:121
985499314,11cyclictest131rcu_preempt20:00:121
9854993117,10cyclictest131rcu_preempt00:10:151
984599314,9cyclictest12-21ksoftirqd/021:05:150
9845993124,4cyclictest30523-21apt-get20:05:130
9845993121,6cyclictest0-21swapper/000:25:150
987499303,19cyclictest321rcuc/320:45:103
987499302,17cyclictest35966irq/58-eth019:45:143
9874993014,10cyclictest35966irq/58-eth000:25:153
986299302,21cyclictest0-21swapper/220:15:132
986299301,17cyclictest27346irq/40-dwc2_hso21:15:122
984599302,22cyclictest22949-21ssh22:55:140
986299294,16cyclictest27346irq/40-dwc2_hso20:05:122
986299293,18cyclictest0-21swapper/222:15:222
986299291,16cyclictest24936-21ssh22:20:132
986299291,14cyclictest930-21snmpd20:45:102
985499292,20cyclictest14068-21grep19:45:221
9854992916,4cyclictest8995-21irqcore23:05:181
985499291,17cyclictest21-21ksoftirqd/123:10:131
984599299,4cyclictest131rcu_preempt22:05:170
984599299,4cyclictest131rcu_preempt22:05:160
984599292,19cyclictest6378-21ntp_states20:10:210
987499285,19cyclictest35966irq/58-eth022:50:133
986299284,14cyclictest27346irq/40-dwc2_hso23:00:112
986299283,8cyclictest0-21swapper/221:50:232
986299282,7cyclictest6943-21ssh22:10:132
986299282,20cyclictest0-21swapper/221:00:102
984599285,8cyclictest12-21ksoftirqd/023:50:230
984599283,4cyclictest12-21ksoftirqd/023:15:280
984599282,21cyclictest28421-21ssh21:10:130
9845992821,3cyclictest26984-21apt-get20:35:130
987499275,19cyclictest35966irq/58-eth021:40:133
986299275,14cyclictest27346irq/40-dwc2_hso00:25:132
986299272,20cyclictest6018-21ssh22:09:232
986299272,20cyclictest6018-21ssh22:09:232
985499274,8cyclictest21-21ksoftirqd/120:51:221
985499272,19cyclictest6717-21cut21:50:231
985499272,19cyclictest13412-21ntp_192.168.11419:10:221
985499272,19cyclictest11141-21ntp_192.168.11420:15:221
9854992718,4cyclictest211ksoftirqd/122:40:121
985499271,18cyclictest10857-21ssh00:00:111
984599275,4cyclictest12-21ksoftirqd/021:25:020
984599272,7cyclictest12-21ksoftirqd/022:30:150
984599272,19cyclictest9888-21awk19:40:220
984599272,19cyclictest9888-21awk19:40:220
986299263,7cyclictest0-21swapper/222:25:232
986299262,9cyclictest31593-21ssh23:52:192
986299262,15cyclictest27346irq/40-dwc2_hso19:30:122
986299261,21cyclictest14038-21sh21:20:152
986299261,14cyclictest10856-21sh00:00:112
985499263,18cyclictest14037-21ssh21:20:151
985499262,20cyclictest1862-21ssh23:55:141
9845992610,9cyclictest14593-21ssh00:20:140
986299253,15cyclictest27346irq/40-dwc2_hso19:35:122
986299252,16cyclictest27346irq/40-dwc2_hso19:45:152
9862992511,8cyclictest27-21ksoftirqd/200:30:142
9862992511,8cyclictest27-21ksoftirqd/200:30:142
986299250,16cyclictest23602-21cut21:05:172
985499253,7cyclictest131rcu_preempt00:08:221
985499252,4cyclictest131rcu_preempt00:20:131
985499252,18cyclictest16100-21apt-get19:50:121
985499251,3cyclictest0-21swapper/123:20:141
984599254,3cyclictest131rcu_preempt19:20:140
984599254,11cyclictest0-21swapper/020:00:120
984599254,11cyclictest0-21swapper/020:00:120
984599253,5cyclictest12-21ksoftirqd/023:35:480
984599253,5cyclictest12-21ksoftirqd/000:34:110
984599253,5cyclictest12-21ksoftirqd/000:34:100
984599252,4cyclictest12-21ksoftirqd/023:40:220
9845992517,3cyclictest17396-21apt-get20:25:110
984599251,6cyclictest12-21ksoftirqd/021:33:430
987499242,18cyclictest35966irq/58-eth019:25:113
987499242,14cyclictest35966irq/58-eth022:35:123
986299246,11cyclictest27346irq/40-dwc2_hso21:55:122
986299242,16cyclictest0-21swapper/220:10:222
986299241,9cyclictest0-21swapper/219:15:132
986299241,14cyclictest930-21snmpd21:30:122
9862992410,10cyclictest27-21ksoftirqd/222:55:132
985499242,17cyclictest9025-21ssh00:35:121
984599247,4cyclictest12-21ksoftirqd/019:30:190
984599243,6cyclictest12-21ksoftirqd/023:00:040
984599243,5cyclictest12-21ksoftirqd/022:04:000
984599243,4cyclictest12-21ksoftirqd/000:15:000
984599242,4cyclictest12-21ksoftirqd/022:35:210
984599242,4cyclictest12-21ksoftirqd/019:15:180
9845992417,3cyclictest16088-21apt-get23:45:120
9845992416,5cyclictest12300-21apt-get21:55:120
984599241,4cyclictest12-21ksoftirqd/022:15:000
9845992413,7cyclictest12560-21apt-get20:20:120
9845992413,7cyclictest12560-21apt-get20:20:110
987499231,11cyclictest35966irq/58-eth021:20:143
986299236,8cyclictest19630-21grep19:50:252
986299232,6cyclictest0-21swapper/220:00:232
986299232,6cyclictest0-21swapper/220:00:232
986299230,13cyclictest13788-21find20:55:112
985499239,4cyclictest6265-21basename19:40:011
985499231,9cyclictest19427-21sh22:35:121
984599237,4cyclictest12-21ksoftirqd/022:40:170
984599234,4cyclictest12-21ksoftirqd/020:30:170
984599234,3cyclictest131rcu_preempt21:15:210
984599233,5cyclictest12-21ksoftirqd/023:15:020
984599233,4cyclictest12-21ksoftirqd/023:35:010
984599232,5cyclictest12-21ksoftirqd/000:05:180
9845992310,9cyclictest12-21ksoftirqd/020:50:140
984599230,6cyclictest12-21ksoftirqd/023:20:130
987499229,9cyclictest0-21swapper/322:05:283
987499229,9cyclictest0-21swapper/322:05:283
987499229,10cyclictest321rcuc/320:15:123
9874992215,4cyclictest0-21swapper/300:15:253
9874992213,6cyclictest0-21swapper/321:26:513
987499221,16cyclictest0-21swapper/321:45:123
987499221,16cyclictest0-21swapper/321:45:123
9874992211,5cyclictest0-21swapper/300:04:483
986299223,13cyclictest27346irq/40-dwc2_hso21:40:132
986299222,15cyclictest0-21swapper/220:35:212
9862992211,8cyclictest0-21swapper/223:05:162
986299221,17cyclictest9115-21fschecks_time20:50:162
986299221,17cyclictest7113-21apt-get23:40:142
986299221,15cyclictest27346irq/40-dwc2_hso20:30:142
9862992210,10cyclictest23950-21kthreadcore19:55:212
985499221,15cyclictest25220-21find23:50:141
984599229,5cyclictest12-21ksoftirqd/000:35:140
984599229,3cyclictest131rcu_preempt20:40:270
984599224,4cyclictest12-21ksoftirqd/022:45:310
984599224,4cyclictest12-21ksoftirqd/022:45:310
984599223,5cyclictest12-21ksoftirqd/000:00:200
984599222,5cyclictest12-21ksoftirqd/019:10:180
987499217,8cyclictest35966irq/58-eth023:00:133
987499212,14cyclictest35966irq/58-eth022:25:163
9874992114,5cyclictest0-21swapper/323:55:143
9874992113,6cyclictest0-21swapper/322:20:443
986299217,11cyclictest30900-21irqcore23:35:182
986299217,11cyclictest29253-21kthreadcore21:10:192
986299211,8cyclictest32268-21kthreadcore00:10:242
986299211,17cyclictest23410-21timerwakeupswit19:20:272
986299211,10cyclictest11149-21kthreadcore00:35:212
9862992110,9cyclictest22510-21ssh23:10:362
985499213,10cyclictest27610-21kworker/1:022:10:371
985499211,16cyclictest32326-21cut21:30:151
985499211,15cyclictest5594-21rm00:15:141
985499211,15cyclictest21440-21sh23:30:121
984599215,5cyclictest12-21ksoftirqd/019:10:020
984599213,6cyclictest12-21ksoftirqd/021:00:020
984599213,4cyclictest12-21ksoftirqd/019:45:230
984599210,9cyclictest4346-21kthreadcore19:35:200
984599210,5cyclictest18807-21tr19:50:200
987499209,8cyclictest0-21swapper/323:14:323
9874992014,4cyclictest0-21swapper/322:46:563
9874992014,4cyclictest0-21swapper/322:46:553
9874992014,4cyclictest0-21swapper/321:35:303
9874992014,4cyclictest0-21swapper/300:12:373
9874992013,5cyclictest0-21swapper/323:40:503
9874992012,5cyclictest0-21swapper/323:50:313
9874992012,5cyclictest0-21swapper/323:30:283
9874992010,8cyclictest0-21swapper/322:40:293
986299201,16cyclictest14442-21grep20:20:202
986299201,16cyclictest14442-21grep20:20:202
986299201,10cyclictest11048-21ps22:30:182
985499207,3cyclictest0-21swapper/122:22:341
9854992014,4cyclictest0-21swapper/123:35:151
985499201,3cyclictest0-21swapper/120:35:021
985499201,2cyclictest131rcu_preempt21:05:211
9854992012,5cyclictest28030-21diskmemload00:30:171
9854992012,5cyclictest28030-21diskmemload00:30:171
9854992012,5cyclictest0-21swapper/119:30:151
9854992012,4cyclictest21-21ksoftirqd/120:10:251
985499201,15cyclictest14187-21ssh22:30:271
985499201,11cyclictest4569-21latency_hist21:15:021
985499200,9cyclictest20926-21grep20:25:251
984599203,4cyclictest12-21ksoftirqd/020:50:010
984599203,11cyclictest0-21swapper/019:25:110
987499193,11cyclictest0-21swapper/320:00:233
987499193,11cyclictest0-21swapper/320:00:233
987499193,10cyclictest0-21swapper/320:55:233
987499193,10cyclictest0-21swapper/320:50:223
987499192,11cyclictest36066irq/59-eth021:10:123
9874991913,4cyclictest0-21swapper/323:17:393
9874991913,4cyclictest0-21swapper/322:10:223
9874991913,4cyclictest0-21swapper/321:53:403
9874991913,4cyclictest0-21swapper/321:15:233
9874991913,4cyclictest0-21swapper/300:34:223
9874991913,4cyclictest0-21swapper/300:34:223
9874991912,5cyclictest0-21swapper/321:55:153
9874991912,5cyclictest0-21swapper/321:30:133
986299195,11cyclictest17278-21kthreadcore00:20:212
986299192,14cyclictest9897-21latency_hist19:10:022
986299191,15cyclictest24582-21fschecks_count19:25:142
985499199,7cyclictest17896-21sed22:50:291
985499198,3cyclictest131rcu_preempt21:35:241
985499197,4cyclictest131rcu_preempt20:50:011
985499197,2cyclictest131rcu_preempt21:15:231
985499191,2cyclictest131rcu_preempt22:55:271
9854991910,6cyclictest6549-21cron22:10:011
9854991910,6cyclictest6549-21cron22:10:011
984599193,4cyclictest12-21ksoftirqd/019:55:210
987499188,8cyclictest0-21swapper/323:07:133
987499182,10cyclictest0-21swapper/319:40:143
987499182,10cyclictest0-21swapper/319:40:143
9874991812,4cyclictest0-21swapper/323:20:173
9874991812,4cyclictest0-21swapper/319:30:213
987499181,13cyclictest0-21swapper/319:40:013
985499188,7cyclictest12349-21apt-get23:25:131
985499185,4cyclictest131rcu_preempt21:25:171
985499184,10cyclictest4862-21apt-get22:45:121
985499184,10cyclictest4862-21apt-get22:45:121
985499184,10cyclictest21-21ksoftirqd/119:15:121
985499182,3cyclictest131rcu_preempt20:35:241
985499182,3cyclictest131rcu_preempt19:25:141
985499181,2cyclictest131rcu_preempt19:40:241
985499181,2cyclictest131rcu_preempt19:40:241
985499180,12cyclictest18592-21find21:00:091
987499177,7cyclictest321rcuc/320:25:113
987499172,9cyclictest0-21swapper/320:05:133
987499172,10cyclictest0-21swapper/320:35:223
987499172,10cyclictest0-21swapper/319:50:223
985499179,6cyclictest21-21ksoftirqd/121:40:191
985499171,7cyclictest15933-21apt22:15:121
9854991711,4cyclictest24305-21kthreadcore00:25:221
985499163,10cyclictest9755-21latency_hist19:10:021
987499159,4cyclictest0-21swapper/320:30:283
987499159,4cyclictest0-21swapper/319:15:103
987499151,10cyclictest0-21swapper/320:20:133
987499151,10cyclictest0-21swapper/320:20:133
985499151,10cyclictest13930-21cut20:55:141
987499131,3cyclictest0-21swapper/319:20:143
987499121,9cyclictest0-21swapper/320:10:153
987499101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional