You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 12:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 21, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1175899614,50cyclictest3773-21ssh22:55:160
11787995337,11cyclictest29566irq/58-eth022:25:143
1177999514,34cyclictest28169-21ssh22:50:142
1178799493,30cyclictest29566irq/58-eth023:55:143
1178799493,30cyclictest29566irq/58-eth023:55:143
1177999442,34cyclictest20472-21ssh22:25:142
1177999438,17cyclictest27-21ksoftirqd/223:55:132
1177999438,17cyclictest27-21ksoftirqd/223:55:132
1177999434,30cyclictest27883-21ssh23:50:152
1177999422,32cyclictest11701-21ssh23:20:152
11768994132,5cyclictest24304-21apt-get20:30:181
11768994131,6cyclictest6468-21apt-get20:45:171
1177999402,17cyclictest932-21snmpd23:45:122
11768994032,5cyclictest30233-21kworker/u8:221:20:161
11768993822,12cyclictest26146-21apt-get00:30:141
1178799374,20cyclictest0-21swapper/322:55:163
1177999378,9cyclictest131rcu_preempt21:15:132
1177999372,13cyclictest11377-21ssh23:40:132
11768993730,4cyclictest1630-21apt-get20:40:151
11768993722,12cyclictest11382-21ssh23:40:131
1175899371,22cyclictest2910-21ssh00:15:160
1177999365,10cyclictest131rcu_preempt22:10:202
1177999363,12cyclictest0-21swapper/222:40:142
1177999362,17cyclictest131rcu_preempt21:40:542
11779993610,19cyclictest11748-21cyclictest00:20:162
11779993610,19cyclictest11748-21cyclictest00:20:152
1176899368,9cyclictest865-21vmstat22:30:341
11768993620,13cyclictest16776-21apt-get19:15:141
1178799358,14cyclictest29566irq/58-eth021:40:133
1177999353,24cyclictest0-21swapper/223:15:552
1176899358,17cyclictest12626-21sh22:20:141
11768993527,5cyclictest21635-21apt-get19:20:121
1176899352,23cyclictest27753-21ssh23:30:141
11768993519,12cyclictest5367-21ssh21:55:151
11768993515,13cyclictest0-21swapper/122:40:141
1175899352,10cyclictest12-21ksoftirqd/020:25:140
1176899347,11cyclictest6945-21tr22:15:221
1176899346,10cyclictest26945-21ssh21:45:421
11768993427,4cyclictest16120-21apt-get20:55:201
11768993411,7cyclictest30249-21diskmemload23:16:131
1178799331,16cyclictest29566irq/58-eth023:30:133
1177999335,18cyclictest26346irq/40-dwc2_hso19:35:162
1176899339,9cyclictest7842-21cat23:35:331
1176899337,8cyclictest0-21swapper/121:36:431
11768993326,3cyclictest27519-21apt-get00:10:151
1176899332,27cyclictest19792-21ssh00:05:141
11768993321,8cyclictest28153-21apt-get22:50:151
11768993310,8cyclictest12137-21rm21:16:401
1175899332,24cyclictest26162-21ssh00:30:150
1177999326,16cyclictest26346irq/40-dwc2_hso20:00:152
1177999325,18cyclictest26346irq/40-dwc2_hso20:40:162
11779993213,11cyclictest932-21snmpd00:05:122
1176899328,8cyclictest23956-21cat23:25:301
1176899328,14cyclictest0-21swapper/123:49:021
1176899327,9cyclictest3867-21ssh00:15:221
1176899327,9cyclictest26368-21rm21:25:251
1176899327,9cyclictest20462-21/usr/sbin/munin22:25:251
1176899327,10cyclictest17477-21sed21:40:231
1176899326,9cyclictest15872-21/usr/sbin/munin00:00:311
1176899322,20cyclictest4036-21ssh22:35:151
11768993215,13cyclictest0-21swapper/123:10:131
1178799314,15cyclictest29566irq/58-eth022:50:133
1178799313,18cyclictest29566irq/58-eth019:25:133
1177999312,11cyclictest261rcuc/219:15:142
1176899316,11cyclictest31076-21kthreadcore20:35:201
11768993125,3cyclictest29865-21apt-get21:50:161
11768993124,4cyclictest3740-21apt-get23:55:151
11768993124,4cyclictest3740-21apt-get23:55:141
11768993119,8cyclictest30612-21ssh21:10:131
11768993116,12cyclictest0-21swapper/120:50:131
1175899312,21cyclictest18001-21ssh23:22:540
1178799306,11cyclictest0-21swapper/323:15:553
1178799302,14cyclictest29566irq/58-eth021:35:133
11787993015,9cyclictest29566irq/58-eth000:00:133
1176899308,8cyclictest23773-21cat23:05:311
1176899308,8cyclictest21732-21mailstats19:50:261
1176899308,14cyclictest19961-21ssh22:03:211
1176899307,8cyclictest24799-21memory22:05:231
1176899307,7cyclictest18071-21ntp_192.168.11420:20:251
1176899306,9cyclictest10714-21ssh22:58:581
1176899306,7cyclictest0-21swapper/119:55:151
1175899306,6cyclictest0-21swapper/023:10:130
1175899302,21cyclictest8666-21ssh23:15:560
1178799295,15cyclictest321rcuc/323:35:163
1177999293,20cyclictest0-21swapper/223:25:142
1177999292,21cyclictest0-21swapper/221:27:422
1177999292,20cyclictest0-21swapper/220:10:172
1177999292,19cyclictest26346irq/40-dwc2_hso19:45:132
1177999291,11cyclictest261rcuc/221:20:152
1176899299,8cyclictest2159-21ssh22:11:241
1176899296,9cyclictest30105-21cat21:05:311
1176899296,9cyclictest30105-21cat21:05:301
1176899296,9cyclictest20431-21ps22:45:181
1176899296,8cyclictest11860-21/usr/sbin/munin23:00:121
1176899295,11cyclictest1072-21ssh23:51:421
1176899293,17cyclictest30589-21ssh21:30:151
11768992921,3cyclictest11703-21apt-get23:20:171
11768992910,15cyclictest21-21ksoftirqd/120:00:141
1175899299,10cyclictest629-21Async14:00:480
1177999281,17cyclictest4752-21ssh22:15:152
1176899285,7cyclictest0-21swapper/100:25:131
1175899283,3cyclictest131rcu_preempt00:10:290
1175899281,21cyclictest12672-21sh22:20:170
1177999274,13cyclictest26346irq/40-dwc2_hso23:00:122
11779992717,3cyclictest11904-21ssh00:00:132
1177999271,20cyclictest12914-21ssh21:38:542
1176899274,10cyclictest16273-21ssh00:22:111
1176899274,10cyclictest16273-21ssh00:22:111
11768992716,7cyclictest11938-21apt-get19:10:161
1175899271,19cyclictest6540-21ssh21:15:150
1177999263,17cyclictest0-21swapper/221:59:562
11779992616,7cyclictest26346irq/40-dwc2_hso00:35:152
1176899265,7cyclictest30249-21diskmemload00:38:241
1176899262,11cyclictest16003-21grep19:45:211
11768992620,3cyclictest0-21swapper/120:05:121
1175899262,18cyclictest4035-21ssh22:35:150
1175899262,18cyclictest29547-21ssh21:28:540
1177999255,11cyclictest26346irq/40-dwc2_hso20:25:152
1177999254,12cyclictest26346irq/40-dwc2_hso20:35:122
1177999252,8cyclictest3139-21ssh23:14:552
1177999251,19cyclictest932-21snmpd20:05:132
1177999251,12cyclictest22026-21grep19:50:282
1175899259,11cyclictest131rcu_preempt20:15:140
1175899257,8cyclictest12-21ksoftirqd/021:38:020
1175899257,8cyclictest12-21ksoftirqd/021:30:180
1175899256,4cyclictest131rcu_preempt21:20:230
1175899255,4cyclictest131rcu_preempt00:25:140
1175899251,19cyclictest23398-21ssh23:25:270
11758992511,8cyclictest12-21ksoftirqd/021:55:010
11758992511,7cyclictest12-21ksoftirqd/022:15:350
1178799243,14cyclictest0-21swapper/323:22:543
11787992412,7cyclictest29566irq/58-eth022:35:273
11787992411,10cyclictest0-21swapper/321:30:213
11779992412,9cyclictest6482-21sh00:15:282
1177999241,16cyclictest26346irq/40-dwc2_hso19:40:102
1176899246,14cyclictest24041-21kthreadcore21:00:231
11768992413,7cyclictest0-21swapper/119:25:121
1175899248,7cyclictest131rcu_preempt23:47:560
1175899246,8cyclictest131rcu_preempt23:05:260
1175899246,3cyclictest131rcu_preempt21:45:140
11758992413,4cyclictest12-21ksoftirqd/022:45:190
11758992412,4cyclictest12-21ksoftirqd/023:34:400
11758992410,8cyclictest12-21ksoftirqd/023:55:070
11758992410,8cyclictest12-21ksoftirqd/023:55:060
11758992410,8cyclictest12-21ksoftirqd/021:10:150
1177999236,11cyclictest26346irq/40-dwc2_hso20:30:192
1177999234,14cyclictest26346irq/40-dwc2_hso21:10:132
1177999234,11cyclictest5149-21cut22:55:212
1177999231,7cyclictest131rcu_preempt21:50:012
11779992317,3cyclictest13911-21ssh22:00:182
1177999231,10cyclictest21275-21ssh22:05:132
1177999230,6cyclictest3655-21ls23:35:202
1176899236,7cyclictest0-21swapper/120:15:191
1175899239,4cyclictest12-21ksoftirqd/020:40:300
1175899234,5cyclictest12-21ksoftirqd/020:45:320
1175899232,8cyclictest131rcu_preempt20:35:010
1175899231,7cyclictest13532-21sh23:40:220
1175899231,16cyclictest14166-21ssh00:20:260
1175899231,16cyclictest14166-21ssh00:20:260
1175899230,4cyclictest131rcu_preempt22:10:180
1178799220,12cyclictest29566irq/58-eth020:15:143
1177999228,11cyclictest941-21kthreadcore21:30:242
1177999227,3cyclictest131rcu_preempt22:20:242
1177999222,15cyclictest0-21swapper/221:53:532
1176899229,11cyclictest11748-21cyclictest19:10:011
11768992215,4cyclictest8496-21apt-get19:40:101
11768992214,5cyclictest22899-21ntp_192.168.11420:25:261
11768992213,6cyclictest5007-21apt-get20:10:171
11768992210,9cyclictest8301-21latency_hist19:40:011
11768992210,9cyclictest3256-21cat19:30:311
1175899229,4cyclictest12-21ksoftirqd/022:30:180
1175899228,5cyclictest12-21ksoftirqd/022:29:500
1175899227,9cyclictest12-21ksoftirqd/022:51:150
1175899225,5cyclictest131rcu_preempt22:00:200
11758992214,5cyclictest11861-21apt-get23:00:120
1175899221,3cyclictest131rcu_preempt19:55:130
1175899221,18cyclictest19842-21cpuspeed00:05:130
11787992111,7cyclictest0-21swapper/300:15:023
1178799211,14cyclictest0-21swapper/322:01:553
1178799211,10cyclictest29566irq/58-eth021:15:123
1177999218,6cyclictest29581-21cut00:10:232
1177999214,6cyclictest0-21swapper/219:10:152
1177999213,16cyclictest11740-21irqcore20:50:182
1177999212,15cyclictest0-21swapper/200:30:142
1177999212,11cyclictest31416-21fschecks_time19:30:172
1177999211,6cyclictest29380-21kthreadcore19:25:232
1177999211,12cyclictest135-21systemd-journal21:10:012
1177999211,12cyclictest135-21systemd-journal21:10:012
1175899219,4cyclictest12-21ksoftirqd/019:32:210
1175899218,4cyclictest12-21ksoftirqd/022:40:270
1175899216,6cyclictest12-21ksoftirqd/020:21:130
1175899214,2cyclictest131rcu_preempt19:25:130
1175899212,4cyclictest12-21ksoftirqd/022:05:150
1175899212,4cyclictest0-21swapper/020:50:290
1175899211,4cyclictest0-21swapper/021:00:290
1175899211,3cyclictest0-21swapper/021:55:240
1175899211,16cyclictest27882-21sh23:50:150
11758992110,3cyclictest131rcu_preempt00:35:210
1178799209,8cyclictest0-21swapper/322:48:423
1178799209,6cyclictest0-21swapper/300:17:153
1178799201,12cyclictest0-21swapper/319:40:013
1177999206,10cyclictest0-21swapper/219:20:142
1177999205,8cyclictest16451-21irqcore20:55:242
1177999202,14cyclictest20094-21diskmemload23:05:192
1177999201,9cyclictest22150-21kthreadcore21:00:202
1177999201,9cyclictest20077-21ssh22:45:142
1177999201,9cyclictest16830-21kthreadcore20:20:212
1177999201,4cyclictest0-21swapper/219:55:142
1177999201,15cyclictest9978-21fschecks_time20:15:182
1177999201,14cyclictest6543-21ls20:45:172
1177999201,11cyclictest22257-21unixbench_singl00:25:302
1177999201,10cyclictest31652-21ls23:30:292
1177999201,10cyclictest29358-21kthreadcore22:30:202
1175899208,4cyclictest12-21ksoftirqd/023:35:110
1175899203,3cyclictest131rcu_preempt20:35:200
1175899202,4cyclictest12-21ksoftirqd/019:50:200
1175899202,3cyclictest131rcu_preempt21:40:140
1175899202,10cyclictest12-21ksoftirqd/020:10:240
1175899201,15cyclictest14336-21kthreadcore00:00:230
1175899201,10cyclictest29540-21grep21:05:300
1175899201,10cyclictest29540-21grep21:05:290
11758992010,4cyclictest12-21ksoftirqd/019:45:270
1175899200,17cyclictest32713-21uniq20:05:150
1178799199,8cyclictest0-21swapper/322:21:243
1178799199,6cyclictest0-21swapper/321:28:543
1178799198,8cyclictest0-21swapper/300:39:093
11787991914,3cyclictest29666irq/59-eth021:57:533
1178799191,12cyclictest0-21swapper/322:40:153
11787991910,6cyclictest29566irq/58-eth021:50:183
11787991910,6cyclictest0-21swapper/300:28:583
1177999191,9cyclictest4202-21grep22:35:172
11758991913,4cyclictest0-21swapper/019:15:150
11758991911,6cyclictest26209-21cron19:25:010
1175899191,13cyclictest12262-21tr19:10:180
1178799189,6cyclictest0-21swapper/323:44:443
1178799189,6cyclictest0-21swapper/321:45:143
1178799189,6cyclictest0-21swapper/300:23:063
1178799189,6cyclictest0-21swapper/300:23:063
1178799189,5cyclictest29566irq/58-eth000:30:153
1178799182,12cyclictest0-21swapper/323:45:143
1178799182,11cyclictest0-21swapper/323:14:553
1178799182,10cyclictest0-21swapper/323:00:043
1178799182,10cyclictest0-21swapper/319:50:143
11787991812,4cyclictest0-21swapper/322:05:233
11787991811,5cyclictest0-21swapper/321:10:173
11787991811,4cyclictest29566irq/58-eth000:05:283
1178799181,11cyclictest0-21swapper/322:10:183
11787991810,6cyclictest0-21swapper/322:33:173
11787991810,6cyclictest0-21swapper/322:16:353
11787991810,6cyclictest0-21swapper/321:24:013
11787991810,5cyclictest29566irq/58-eth023:50:143
1175899182,4cyclictest12-21ksoftirqd/019:40:090
1175899182,13cyclictest5206-21kthreadcore19:35:210
1178799179,6cyclictest0-21swapper/323:28:453
1178799178,6cyclictest33-21ksoftirqd/323:05:373
1178799171,12cyclictest0-21swapper/320:55:013
1175899174,3cyclictest131rcu_preempt20:00:180
1178799165,8cyclictest29566irq/58-eth020:35:133
1178799162,9cyclictest0-21swapper/320:10:183
1178799162,8cyclictest0-21swapper/321:00:133
1175899160,2cyclictest131rcu_preempt19:10:010
1178799149,3cyclictest0-21swapper/320:45:163
1178799149,3cyclictest0-21swapper/320:40:253
1178799149,3cyclictest0-21swapper/320:25:423
1178799146,5cyclictest0-21swapper/319:20:113
1178799142,10cyclictest0-21swapper/319:45:133
1178799138,3cyclictest0-21swapper/320:00:223
1178799135,6cyclictest0-21swapper/320:30:263
1178799131,10cyclictest0-21swapper/321:05:163
1178799131,10cyclictest0-21swapper/321:05:163
1178799131,10cyclictest0-21swapper/319:55:293
1177999139,2cyclictest261rcuc/219:10:022
1178799128,3cyclictest0-21swapper/320:59:123
1178799124,5cyclictest0-21swapper/319:30:153
1178799122,7cyclictest0-21swapper/320:20:153
1178799122,5cyclictest0-21swapper/319:10:153
1178799121,9cyclictest0-21swapper/319:40:183
1178799115,4cyclictest29566irq/58-eth019:15:143
1178799111,7cyclictest0-21swapper/320:05:183
117879991,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional