You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 03:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Dec 06, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8820997444,25cyclictest29566irq/58-eth023:35:133
880899591,7cyclictest0-21swapper/220:50:132
880099586,8cyclictest131rcu_preempt20:30:151
878999505,38cyclictest22738-21ssh23:05:140
880899483,29cyclictest21356-21ssh21:45:132
880099482,37cyclictest22409-21ssh23:25:171
878999481,43cyclictest24731-21apt-get20:00:130
878999481,37cyclictest14429-21ssh23:20:170
880099473,35cyclictest14154-21ssh22:00:121
878999471,42cyclictest3271-21apt-get20:45:100
878999471,41cyclictest17773-21apt-get21:00:150
878999464,31cyclictest6979-21ssh22:15:170
878999464,31cyclictest6979-21ssh22:15:170
878999461,42cyclictest0-21swapper/020:55:150
878999461,42cyclictest0-21swapper/020:55:140
880099456,29cyclictest12941-21ssh21:40:111
878999451,41cyclictest15055-21apt-get22:20:160
878999451,38cyclictest6789-21apt-get20:15:140
880099442,34cyclictest28576-21ssh21:30:131
878999441,38cyclictest3376-21apt-get21:15:130
878999441,38cyclictest27523-21apt-get21:10:140
878999441,37cyclictest5816-21apt-get23:55:140
878999441,37cyclictest0-21swapper/021:25:150
878999440,40cyclictest15064-21apt-get19:50:130
878999421,37cyclictest31229-21apt-get22:50:140
878999421,37cyclictest0-21swapper/020:30:140
878999421,35cyclictest14151-21apt-get22:00:130
878999421,26cyclictest14986-21df22:40:180
878999420,30cyclictest5344-21ssh23:54:590
878999420,27cyclictest22303-21ssh23:25:100
878999420,26cyclictest2317-21ssh21:32:270
878999411,36cyclictest0-21swapper/022:30:140
878999410,37cyclictest22586-21apt-get21:05:150
882099403,21cyclictest29566irq/58-eth021:40:113
880899402,22cyclictest22185-21sh23:04:412
878999401,34cyclictest0-21swapper/021:55:130
878999401,34cyclictest0-21swapper/020:10:150
878999391,23cyclictest0-21swapper/021:40:150
880899382,29cyclictest19448-21ssh23:41:432
880899382,28cyclictest5030-21ssh22:52:402
880899381,30cyclictest21912-21ssh23:45:142
880099381,32cyclictest31233-21ssh22:50:131
878999381,34cyclictest30886-21ssh22:10:150
878999381,34cyclictest23162-21apt-get22:25:170
878999381,34cyclictest2070-21apt-get00:35:140
878999381,34cyclictest0-21swapper/019:35:130
878999380,26cyclictest11615-21ssh21:38:400
880099373,26cyclictest29839-21ssh21:50:121
878999374,29cyclictest0-21swapper/000:10:130
878999371,32cyclictest0-21swapper/020:05:110
878999371,31cyclictest0-21swapper/023:45:140
878999371,30cyclictest0-21swapper/000:00:140
878999371,24cyclictest11594-21ssh00:18:300
878999371,24cyclictest11594-21ssh00:18:290
878999370,28cyclictest28099-21ssh22:06:380
878999370,22cyclictest3892-21sh21:52:480
878999361,31cyclictest6911-21apt-get22:55:140
878999361,31cyclictest20548-21ssh00:25:150
882099353,16cyclictest29566irq/58-eth021:10:143
880899352,21cyclictest14413-21ssh23:20:162
880099352,28cyclictest0-21swapper/122:45:121
878999351,20cyclictest1-21systemd23:12:060
878999351,20cyclictest1-21systemd23:12:050
880899343,21cyclictest31053-21ssh22:30:142
880899343,17cyclictest26346irq/40-dwc2_hso23:50:122
878999341,30cyclictest2117-21cat23:30:310
878999341,30cyclictest0-21swapper/000:20:120
878999341,29cyclictest9662-21cut19:10:200
878999341,29cyclictest0-21swapper/022:46:440
878999341,29cyclictest0-21swapper/020:20:130
882099332,16cyclictest29566irq/58-eth023:25:163
878999331,28cyclictest16700-21/usr/sbin/munin20:25:190
878999331,27cyclictest0-21swapper/021:20:140
878999330,30cyclictest8520-21kthreadcore22:35:210
878999330,29cyclictest10096-21/usr/sbin/munin23:15:290
878999330,21cyclictest27332-21sshd00:06:410
878999321,28cyclictest0-21swapper/023:00:130
878999321,27cyclictest13719-21apt-get19:15:120
878999321,27cyclictest0-21swapper/019:55:120
878999321,26cyclictest0-21swapper/019:20:120
878999321,23cyclictest21339-21apt-get21:45:110
878999320,29cyclictest26712-21irqstats20:35:180
880899315,19cyclictest5922-21ssh23:35:122
8800993123,4cyclictest0-21swapper/120:45:101
8789993115,13cyclictest530-21usb-storage19:25:120
878999311,27cyclictest8205-21/usr/sbin/munin20:50:170
878999311,26cyclictest5924-21apt-get23:35:140
880899302,17cyclictest30317-21ssh23:30:142
880899302,16cyclictest6928-21ssh22:55:152
880099302,22cyclictest6594-21ssh22:35:151
8800993016,7cyclictest0-21swapper/121:00:131
878999301,26cyclictest5403-21munin-node19:40:120
878999301,26cyclictest31517-21unixbench_multi00:30:320
878999301,26cyclictest31020-21df_inode20:40:150
878999301,20cyclictest0-21swapper/019:45:130
878999300,28cyclictest20067-21ssh23:42:480
882099295,16cyclictest29566irq/58-eth022:45:113
882099294,16cyclictest29566irq/58-eth022:50:153
880899292,20cyclictest0-21swapper/222:40:162
880099292,17cyclictest6917-21ssh22:55:131
878999291,24cyclictest0-21swapper/019:30:130
878999290,26cyclictest8821-21latency_hist19:10:020
878999290,26cyclictest8821-21latency_hist19:10:010
882099282,18cyclictest29566irq/58-eth021:30:133
882099282,14cyclictest29566irq/58-eth021:35:153
880899288,14cyclictest29851-21diskmemload21:50:142
880899283,20cyclictest0-21swapper/222:12:402
880899282,20cyclictest0-21swapper/219:15:142
880899282,19cyclictest0-21swapper/220:35:142
880899282,16cyclictest26346irq/40-dwc2_hso21:55:122
880899281,18cyclictest0-21swapper/223:05:132
880899281,14cyclictest5048-21ssh00:15:152
880899281,14cyclictest5048-21ssh00:15:142
8820992720,5cyclictest29566irq/58-eth000:25:153
880899277,12cyclictest27-21ksoftirqd/221:00:242
880899275,13cyclictest26346irq/40-dwc2_hso22:25:182
880899271,17cyclictest23209-21ls19:25:012
880899266,14cyclictest26346irq/40-dwc2_hso22:00:132
8808992610,8cyclictest27-21ksoftirqd/223:19:202
8808992610,6cyclictest27-21ksoftirqd/221:33:032
880099263,18cyclictest22499-21ssh22:05:151
880099262,18cyclictest19809-21ssh23:00:411
880099262,18cyclictest19797-21ssh22:40:401
880099261,16cyclictest27537-21ssh21:10:131
882099252,16cyclictest0-21swapper/322:55:143
880899259,9cyclictest131rcu_preempt21:15:532
880899256,13cyclictest27-21ksoftirqd/222:23:302
880899255,12cyclictest26346irq/40-dwc2_hso22:05:132
880899252,16cyclictest26346irq/40-dwc2_hso20:10:162
880899252,16cyclictest26346irq/40-dwc2_hso20:05:122
8808992514,4cyclictest0-21swapper/221:13:432
880899250,16cyclictest14059-21ls00:00:172
880099258,12cyclictest21-21ksoftirqd/123:41:421
880099254,14cyclictest27168-21diskmemload23:55:151
882099242,14cyclictest0-21swapper/322:00:133
882099241,14cyclictest0-21swapper/322:40:223
880899248,7cyclictest131rcu_preempt00:06:152
8808992413,9cyclictest0-21swapper/200:20:122
880899241,18cyclictest29336-21ssh22:47:412
882099233,13cyclictest0-21swapper/323:01:423
8820992315,5cyclictest29566irq/58-eth000:35:143
882099231,14cyclictest29566irq/58-eth023:55:153
880899239,4cyclictest27-21ksoftirqd/221:25:022
880899235,15cyclictest25965-21sh23:25:262
880899233,13cyclictest27-21ksoftirqd/223:55:562
880899232,14cyclictest27-21ksoftirqd/221:40:122
8808992310,6cyclictest131rcu_preempt00:39:012
880099233,16cyclictest31052-21ssh22:30:131
880099231,9cyclictest21440-21apt-get23:45:021
8800992314,5cyclictest0-21swapper/121:35:151
880099231,18cyclictest10299-21ssh23:15:301
8800992310,4cyclictest131rcu_preempt23:05:171
880899225,12cyclictest842-21munin-node00:30:152
880899225,10cyclictest27-21ksoftirqd/222:15:452
880899225,10cyclictest27-21ksoftirqd/222:15:442
8808992212,8cyclictest0-21swapper/220:55:202
8808992212,8cyclictest0-21swapper/220:55:192
8808992212,8cyclictest0-21swapper/220:40:152
8808992210,4cyclictest7716-21kthreadcore21:35:222
880099223,13cyclictest201rcuc/122:25:171
882099213,12cyclictest0-21swapper/322:10:153
880899219,3cyclictest27-21ksoftirqd/221:05:272
880899215,8cyclictest27-21ksoftirqd/219:50:292
880899214,8cyclictest25052-21kthreadcore19:25:202
880899213,4cyclictest27-21ksoftirqd/219:45:012
880899213,15cyclictest3464-21fschecks_time20:45:142
880899212,11cyclictest29090-21kthreadcore19:30:192
8808992111,8cyclictest0-21swapper/221:20:272
880899211,16cyclictest9425-21kthreadcore22:35:222
880899210,9cyclictest27-21ksoftirqd/200:13:442
880099213,9cyclictest1-21systemd20:00:131
880099212,14cyclictest6245-21ssh22:14:391
880099211,2cyclictest131rcu_preempt21:55:131
880099211,16cyclictest0-21swapper/123:10:131
880099211,16cyclictest0-21swapper/123:10:131
882099206,9cyclictest0-21swapper/323:40:233
882099202,12cyclictest0-21swapper/321:50:143
880899207,4cyclictest27-21ksoftirqd/223:14:072
880899207,4cyclictest27-21ksoftirqd/223:14:062
880899207,2cyclictest131rcu_preempt00:28:322
880899205,3cyclictest131rcu_preempt19:10:232
880899205,12cyclictest24950-21wc20:00:172
880899205,12cyclictest1722-21kthreadcore19:35:202
880899202,8cyclictest0-21swapper/220:20:142
880899202,3cyclictest131rcu_preempt20:30:202
880899202,12cyclictest26346irq/40-dwc2_hso19:55:122
880899201,8cyclictest0-21swapper/220:25:162
880899200,17cyclictest10557-21wc20:15:292
880099209,8cyclictest28302-21df19:30:151
880099207,8cyclictest0-21swapper/119:50:131
880099206,4cyclictest131rcu_preempt23:30:151
8800992012,5cyclictest28645-21sh00:30:231
8800992012,5cyclictest0-21swapper/122:20:281
8800992011,5cyclictest211ksoftirqd/121:45:121
880099201,14cyclictest23754-21ps19:25:161
8800992010,8cyclictest18954-21cat23:20:341
8800992010,7cyclictest5596-21netstat00:35:241
882099198,9cyclictest0-21swapper/321:59:543
882099198,7cyclictest29566irq/58-eth023:50:233
882099198,6cyclictest0-21swapper/323:30:133
882099193,13cyclictest0-21swapper/322:05:153
882099193,11cyclictest0-21swapper/323:07:043
882099193,11cyclictest0-21swapper/322:30:143
8820991911,5cyclictest29566irq/58-eth000:20:123
882099191,13cyclictest0-21swapper/320:45:133
882099191,12cyclictest0-21swapper/322:20:203
8820991910,6cyclictest0-21swapper/323:15:153
8820991910,6cyclictest0-21swapper/321:15:123
8820991910,5cyclictest0-21swapper/322:25:233
8808991910,3cyclictest10434-21fschecks_time19:45:172
880099195,11cyclictest12775-21head19:10:291
880099192,12cyclictest29727-21ssh00:10:141
880099191,7cyclictest27163-21munin-run00:30:021
8800991914,3cyclictest0-21swapper/100:20:121
8800991913,4cyclictest530-21usb-storage22:15:051
8800991913,4cyclictest530-21usb-storage22:15:041
880099191,15cyclictest10163-21ls20:15:231
880099191,14cyclictest25065-21tr21:05:231
880099191,14cyclictest17975-21ssh00:00:311
882099189,6cyclictest29566irq/58-eth023:20:153
882099189,6cyclictest0-21swapper/323:11:043
882099189,6cyclictest0-21swapper/323:11:033
882099189,6cyclictest0-21swapper/321:23:383
882099189,6cyclictest0-21swapper/300:31:103
882099189,6cyclictest0-21swapper/300:17:153
882099189,6cyclictest0-21swapper/300:17:143
882099182,10cyclictest0-21swapper/319:15:133
882099182,10cyclictest0-21swapper/319:10:153
8820991812,4cyclictest0-21swapper/320:35:263
882099181,13cyclictest0-21swapper/319:50:293
882099181,13cyclictest0-21swapper/300:10:023
882099181,12cyclictest0-21swapper/320:40:173
882099181,10cyclictest0-21swapper/321:50:023
8820991810,6cyclictest0-21swapper/323:45:433
8820991810,5cyclictest29566irq/58-eth021:25:163
880899181,8cyclictest8823-21latency_hist19:10:022
880899181,8cyclictest8823-21latency_hist19:10:012
880099189,6cyclictest1171-21missed_timers23:50:221
880099181,9cyclictest20061-21swap20:25:291
880099181,8cyclictest20512-21grep21:25:201
882099179,6cyclictest0-21swapper/322:36:403
882099179,6cyclictest0-21swapper/322:19:263
882099179,6cyclictest0-21swapper/322:19:253
882099179,6cyclictest0-21swapper/300:00:163
882099179,5cyclictest29566irq/58-eth000:10:133
882099172,10cyclictest0-21swapper/321:05:163
882099172,10cyclictest0-21swapper/320:30:173
882099171,12cyclictest0-21swapper/320:00:023
882099171,10cyclictest0-21swapper/320:15:153
880099179,5cyclictest0-21swapper/120:35:231
880099178,6cyclictest20995-21cut19:20:211
880099177,8cyclictest1-21systemd20:41:351
880099177,7cyclictest29541-21apt-get20:05:111
880099175,9cyclictest1493-21kthreadcore19:35:191
880099173,11cyclictest5749-21irqcore19:40:191
880099173,11cyclictest25434-21grep00:05:231
880099172,11cyclictest5935-21ssh23:35:151
8800991711,4cyclictest0-21swapper/119:15:241
880099171,13cyclictest9313-21kthreadcore20:50:211
880099171,13cyclictest5028-21apt-get00:15:131
880099171,13cyclictest5028-21apt-get00:15:131
880099171,13cyclictest2415-21irqcore20:10:221
880099171,13cyclictest14887-21cat20:20:231
880099171,12cyclictest3380-21ssh21:15:141
880099162,11cyclictest20928-21kthreadcore19:55:181
880099161,7cyclictest12482-21kernelversion21:20:181
880099161,12cyclictest13266-21irqcore20:55:191
880099161,12cyclictest13266-21irqcore20:55:191
880099161,11cyclictest10609-21irqcore19:45:181
882099159,4cyclictest0-21swapper/320:20:183
882099141,11cyclictest0-21swapper/319:40:293
880099142,10cyclictest8670-21latency_hist19:10:021
880099142,10cyclictest8670-21latency_hist19:10:021
882099138,3cyclictest0-21swapper/321:00:253
882099138,3cyclictest0-21swapper/320:52:253
882099138,3cyclictest0-21swapper/320:10:223
882099133,7cyclictest0-21swapper/321:00:013
882099133,7cyclictest0-21swapper/321:00:003
882099132,7cyclictest0-21swapper/320:25:163
882099131,9cyclictest0-21swapper/319:25:133
882099131,8cyclictest0-21swapper/319:45:133
882099122,7cyclictest0-21swapper/320:05:223
882099122,7cyclictest0-21swapper/319:35:133
882099122,7cyclictest0-21swapper/319:30:173
882099121,9cyclictest0-21swapper/319:20:163
882099121,8cyclictest0-21swapper/319:10:023
882099121,8cyclictest0-21swapper/319:10:013
882099121,7cyclictest0-21swapper/320:05:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional