You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-21 - 04:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue Oct 21, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
556899601,18cyclictest1875466ssh22:10:162
556899563,4cyclictest0-21swapper/220:45:132
556899551,3cyclictest0-21swapper/221:05:152
556899501,43cyclictest0-21swapper/223:25:182
554599492,37cyclictest2729-21ssh22:00:160
556899473,25cyclictest2709-21ssh22:00:152
556899461,6cyclictest0-21swapper/219:30:152
556899461,6cyclictest0-21swapper/219:30:152
558099455,30cyclictest25366irq/58-eth000:00:153
558099455,30cyclictest25366irq/58-eth000:00:143
556899452,4cyclictest0-21swapper/220:55:142
556899442,36cyclictest21846irq/40-dwc2_hso20:20:152
556899441,3cyclictest0-21swapper/220:15:162
558099422,26cyclictest25366irq/58-eth023:20:153
555799412,36cyclictest10837-21ssh22:05:161
555799412,27cyclictest26216-21ssh22:55:161
556899403,2cyclictest0-21swapper/221:55:172
556899402,6cyclictest17592-21apt-get23:50:162
555799400,29cyclictest25716-21ssh23:55:131
556899391,4cyclictest0-21swapper/223:15:132
556899391,2cyclictest8099-21apt-get21:20:112
556899383,27cyclictest0-21swapper/220:50:152
556899382,29cyclictest942-21snmpd21:40:092
556899381,7cyclictest942-21snmpd22:30:192
556899381,5cyclictest25676-21/usr/sbin/munin23:55:312
556899380,7cyclictest24170-21apt-get21:30:132
5557993831,3cyclictest21-21ksoftirqd/121:10:121
558099371,20cyclictest25366irq/58-eth022:30:143
556899371,4cyclictest10682-21/usr/sbin/munin19:15:182
558099364,19cyclictest25366irq/58-eth022:10:153
556899361,9cyclictest30632-21ssh22:15:282
556899361,7cyclictest24192-21latency_hist00:15:022
556899361,7cyclictest23444-21kthreadcore20:35:212
556899360,5cyclictest10101-21ssh22:45:142
555799363,25cyclictest0-21swapper/123:45:141
554599366,23cyclictest12-21ksoftirqd/019:20:180
558099351,21cyclictest25366irq/58-eth023:05:153
556899351,8cyclictest4564-21kthreadcore22:40:232
556899351,7cyclictest1444-21kthreadcore00:20:242
556899351,6cyclictest11807-21sort19:50:142
556899351,6cyclictest11075-21kthreadcore23:45:212
556899351,31cyclictest29346-21tr20:40:212
556899351,31cyclictest1316-21apt23:00:002
556899351,30cyclictest21151-21needreboot23:10:262
556899351,30cyclictest11697-21ssh21:45:202
556899351,11cyclictest24076-21df_inode21:10:152
558099342,16cyclictest25366irq/58-eth023:30:133
556899345,21cyclictest0-21swapper/222:20:292
556899344,3cyclictest27778-21kthreadcore22:35:232
556899341,8cyclictest4888-21ntp_states23:40:282
556899341,7cyclictest29471-21ntp_192.168.11420:05:242
556899341,7cyclictest136-21systemd-journal20:12:442
556899341,6cyclictest5337-21grep23:20:332
556899341,6cyclictest4130-21cut00:00:242
556899341,6cyclictest4130-21cut00:00:232
556899341,6cyclictest3130166ssh00:19:142
556899341,5cyclictest5293-21/usr/sbin/munin23:00:272
5545993414,12cyclictest5533-21cyclictest19:40:110
558099332,16cyclictest25366irq/58-eth019:45:193
558099331,15cyclictest25366irq/58-eth022:40:143
556899331,8cyclictest19293-21ssh21:50:192
556899331,8cyclictest19293-21ssh21:50:182
556899331,7cyclictest18017-21ssh22:50:052
556899331,6cyclictest17566-21ssh22:29:122
556899331,5cyclictest5429-21sort19:40:212
556899331,4cyclictest18908-21cat19:20:302
556899331,4cyclictest13927-21latency_hist00:30:012
556899331,29cyclictest18353-21kthreadcore23:30:212
556899330,5cyclictest23611-21diskmemload22:07:092
555799331,29cyclictest1248-21ssh23:40:151
554599331,19cyclictest26839-21ssh21:55:170
558099323,16cyclictest0-21swapper/323:40:153
5580993211,15cyclictest321rcuc/320:50:153
556899329,20cyclictest10479-21ssh23:05:212
556899321,9cyclictest16471-21find19:55:152
556899321,8cyclictest9650-21tail00:05:182
556899321,8cyclictest16749-21tail20:25:302
556899321,8cyclictest15737-21kthreadcore00:30:202
556899321,5cyclictest5702-21cpu19:10:132
556899321,5cyclictest28052-21ssh00:39:082
556899320,6cyclictest491-21ssh21:36:452
555799324,25cyclictest25519-21ssh23:35:141
556899314,18cyclictest21846irq/40-dwc2_hso20:30:162
556899311,8cyclictest32557-21sh21:15:212
556899311,8cyclictest25369-21sh23:35:042
556899311,5cyclictest29705-21/usr/sbin/munin19:35:152
556899311,3cyclictest6988-21df19:45:192
556899310,5cyclictest21299-21ssh21:26:062
554599311,25cyclictest25635-21ssh23:15:140
556899301,8cyclictest22075-21kthreadcore20:00:182
556899301,7cyclictest21892-21kthreadcore19:25:202
556899301,3cyclictest0-21swapper/221:00:142
554599301,12cyclictest17358-21sh23:30:140
5557992916,10cyclictest21-21ksoftirqd/119:50:111
554599291,19cyclictest18134-21diskmemload22:50:130
558099284,14cyclictest25366irq/58-eth020:15:143
5580992812,11cyclictest25366irq/58-eth021:40:093
556899281,7cyclictest5533-21cyclictest19:10:012
556899281,7cyclictest5533-21cyclictest19:10:012
555799282,16cyclictest1711-21ssh23:00:141
5580992714,8cyclictest25366irq/58-eth000:10:163
555799279,16cyclictest9595-21ssh23:05:141
555799273,18cyclictest17600-21ssh23:10:151
555799272,17cyclictest8110-21ssh21:20:131
554599272,20cyclictest1300-21ssh23:20:160
558099264,17cyclictest0-21swapper/322:45:143
555799261,17cyclictest0-21swapper/119:45:171
558099251,16cyclictest25366irq/58-eth023:50:143
558099251,16cyclictest0-21swapper/322:25:163
555799254,13cyclictest5444-21kworker/u8:023:50:141
555799252,19cyclictest18444-21sh22:30:151
5557992515,7cyclictest136-21systemd-journal00:30:021
555799251,19cyclictest26498-21ssh22:35:171
555799244,17cyclictest17031-21kworker/1:222:10:141
555799244,12cyclictest20410-21kworker/1:000:35:151
555799241,19cyclictest3355-21cut20:15:171
554599241,17cyclictest29568-21apt-key19:35:020
554599241,17cyclictest29568-21apt-key19:35:010
558099232,13cyclictest25366irq/58-eth022:50:133
5580992312,8cyclictest25366irq/58-eth023:25:173
558099231,13cyclictest25366irq/58-eth000:05:123
558099231,11cyclictest25366irq/58-eth000:20:153
5557992312,8cyclictest10096-21cron22:25:011
555799231,13cyclictest9266-21ssh00:05:121
555799222,16cyclictest16758-21ssh00:10:151
5557992212,7cyclictest19768-21kthreadcore23:30:221
554599221,17cyclictest13004-21cut20:25:180
558099218,7cyclictest25366irq/58-eth023:45:153
5557992112,6cyclictest0-21swapper/121:45:151
555799211,15cyclictest23611-21diskmemload23:20:161
554599213,15cyclictest0-21swapper/022:40:140
554599211,7cyclictest23482-21tail00:35:210
558099207,6cyclictest25466irq/59-eth023:10:133
5580992011,6cyclictest25366irq/58-eth021:10:133
558099201,13cyclictest0-21swapper/322:20:023
558099201,10cyclictest25366irq/58-eth021:55:183
5580992010,8cyclictest0-21swapper/322:37:353
5580992010,8cyclictest0-21swapper/321:52:573
5580992010,8cyclictest0-21swapper/321:52:563
5580992010,7cyclictest0-21swapper/323:59:503
555799209,8cyclictest13983-21cat23:25:351
555799202,14cyclictest32205-21ssh21:15:151
555799202,13cyclictest10465-21awk22:05:011
555799201,16cyclictest27574-21tr23:15:201
555799201,10cyclictest20194-21sort19:25:161
5557992010,7cyclictest19948-21mailstats21:25:291
554599202,15cyclictest23932-21find21:10:110
554599202,14cyclictest10958-21ssh21:45:130
554599201,5cyclictest52846usb-storage20:45:050
554599201,17cyclictest18315-21kthreadcore23:10:200
554599201,16cyclictest10585-21ls22:25:180
558099199,8cyclictest0-21swapper/321:18:313
5580991911,5cyclictest0-21swapper/300:35:003
558099191,13cyclictest25366irq/58-eth023:35:153
558099191,13cyclictest0-21swapper/323:00:313
558099191,13cyclictest0-21swapper/319:50:133
558099191,12cyclictest0-21swapper/320:05:153
5580991910,7cyclictest0-21swapper/300:16:163
5580991910,6cyclictest0-21swapper/321:25:143
5580991910,6cyclictest0-21swapper/300:25:433
555799193,9cyclictest0-21swapper/120:10:141
555799193,10cyclictest0-21swapper/122:45:141
555799191,9cyclictest15483-21fschecks_time19:20:191
5557991913,3cyclictest8849-21cut19:10:211
5557991913,3cyclictest21-21ksoftirqd/121:50:311
5557991913,3cyclictest21-21ksoftirqd/121:50:301
555799191,15cyclictest24448-21grep21:30:181
555799191,15cyclictest18021-21sed20:30:201
555799191,15cyclictest13051-21cut20:25:171
555799191,14cyclictest21274-21apt-get20:00:131
554599191,14cyclictest11675-21apt-get19:50:110
554599191,10cyclictest0-21swapper/022:16:420
5545991910,6cyclictest20263-21unixbench_singl21:25:300
554599190,3cyclictest131rcu_preempt20:20:150
554599190,15cyclictest10970-21ls00:25:310
558099189,6cyclictest0-21swapper/323:15:583
558099189,6cyclictest0-21swapper/322:06:463
558099189,6cyclictest0-21swapper/321:47:513
558099189,6cyclictest0-21swapper/321:20:123
558099182,10cyclictest0-21swapper/321:05:153
558099182,10cyclictest0-21swapper/320:30:163
558099182,10cyclictest0-21swapper/320:20:043
558099182,10cyclictest0-21swapper/319:55:163
558099182,10cyclictest0-21swapper/300:35:043
5580991811,5cyclictest25366irq/58-eth022:55:163
558099181,13cyclictest0-21swapper/320:25:133
5580991810,6cyclictest25366irq/58-eth022:20:133
5580991810,6cyclictest0-21swapper/321:31:323
555799185,10cyclictest17908-21grep21:00:281
555799183,8cyclictest5533-21cyclictest22:40:131
555799181,4cyclictest52846usb-storage20:35:111
5557991812,4cyclictest6881-21switchtime21:40:261
555799181,14cyclictest3250-21sh00:20:271
5557991810,6cyclictest31495-21sshd22:15:391
5557991810,5cyclictest18872-21kthreadcore22:50:191
5557991810,5cyclictest15864-21kthreadcore00:30:191
554599189,6cyclictest20497-21uname00:10:300
554599188,3cyclictest17256-21sh22:08:090
554599182,2cyclictest0-21swapper/023:49:270
554599182,12cyclictest1739-21ssh23:00:160
5545991812,4cyclictest0-21swapper/020:40:150
5545991812,4cyclictest0-21swapper/019:15:240
5545991812,3cyclictest0-21swapper/000:00:140
5545991812,3cyclictest0-21swapper/000:00:140
554599181,15cyclictest29893-21sendmail_mailqu22:55:290
5545991811,5cyclictest24107-21ssh22:11:270
554599181,14cyclictest13032-21ls23:05:240
554599181,10cyclictest5065-21ls23:40:290
5545991810,6cyclictest22751-21fw_packets20:35:180
5545991810,6cyclictest18793-21sshd21:00:370
5545991810,6cyclictest1-21systemd22:38:080
5545991810,6cyclictest0-21swapper/023:25:310
558099179,6cyclictest0-21swapper/321:38:273
558099178,5cyclictest0-21swapper/322:00:173
558099172,9cyclictest0-21swapper/320:45:043
558099172,9cyclictest0-21swapper/319:40:123
558099172,9cyclictest0-21swapper/319:25:133
558099172,10cyclictest0-21swapper/319:30:173
558099172,10cyclictest0-21swapper/319:30:163
558099171,12cyclictest0-21swapper/321:05:023
555799179,4cyclictest444-21irqrtprio20:45:181
555799177,6cyclictest11201-21irqrtprio22:25:231
555799172,12cyclictest30625-21apt20:10:011
5557991711,4cyclictest0-21swapper/119:35:151
555799171,13cyclictest7760-21apt-get20:20:041
555799171,13cyclictest27005-21fschecks_count21:55:211
555799171,13cyclictest21053-21sendmail-msp20:00:011
555799171,12cyclictest2669-21munin-run21:40:011
555799171,11cyclictest29410-21apt-get19:35:011
555799171,11cyclictest29410-21apt-get19:35:001
555799171,10cyclictest25276-21kthreadcore00:15:211
554599178,7cyclictest4127-21cat21:15:320
554599173,12cyclictest24615-21cron19:30:010
554599173,10cyclictest0-21swapper/020:55:160
554599172,9cyclictest32112-21ssh00:20:160
554599172,11cyclictest0-21swapper/020:50:150
554599172,11cyclictest0-21swapper/020:30:160
554599172,10cyclictest0-21swapper/020:00:130
554599171,7cyclictest10601-21kthreadcore21:20:200
554599171,6cyclictest18671-21perl22:30:180
554599171,14cyclictest31012-21sort20:10:150
554599171,13cyclictest2468-21ssh22:20:140
554599171,13cyclictest22065-21tr21:05:230
554599171,12cyclictest19851-21kthreadcore21:50:190
554599171,12cyclictest19851-21kthreadcore21:50:190
5545991710,4cyclictest3733-21kthreadcore21:40:170
554599170,15cyclictest27603-21ls21:30:240
554599170,14cyclictest18262-21timerwakeupswit00:30:280
558099162,9cyclictest0-21swapper/320:00:133
558099162,9cyclictest0-21swapper/319:20:183
555799167,6cyclictest9601-21grep20:55:171
555799162,11cyclictest7051-21kthreadcore20:50:241
555799162,11cyclictest29585-21kthreadcore20:40:221
555799162,11cyclictest2226-21fschecks_count19:40:141
555799162,11cyclictest11217-21kthreadcore19:15:201
555799161,12cyclictest1343-21sh00:00:151
555799161,12cyclictest1343-21sh00:00:151
555799161,11cyclictest19281-21idleruntime21:05:181
554599169,4cyclictest0-21swapper/023:35:030
554599167,6cyclictest18963-21cut19:55:240
554599166,7cyclictest29804-21df19:35:170
554599164,9cyclictest6201-21ps19:10:180
554599162,10cyclictest0-21swapper/020:15:160
554599161,12cyclictest29396-21ssh23:55:250
554599161,12cyclictest18302-21irqstats23:50:200
554599161,11cyclictest10203-21/usr/sbin/munin22:45:150
558099152,9cyclictest0-21swapper/320:40:053
558099151,8cyclictest0-21swapper/320:35:283
554599156,7cyclictest10351-21cut00:05:190
554599153,9cyclictest0-21swapper/019:45:180
554599151,11cyclictest29244-21tr20:05:230
554599151,11cyclictest24818-21irqrtprio00:15:180
558099149,3cyclictest0-21swapper/320:10:133
558099141,11cyclictest0-21swapper/320:55:203
558099141,10cyclictest0-21swapper/319:15:203
555799141,10cyclictest5567-21latency_hist19:10:011
555799141,10cyclictest5567-21latency_hist19:10:011
554599148,4cyclictest23611-21diskmemload21:35:340
554599141,10cyclictest5547-21latency_hist19:10:020
554599141,10cyclictest5547-21latency_hist19:10:010
558099138,3cyclictest0-21swapper/319:38:293
558099130,11cyclictest33-21ksoftirqd/319:10:163
558099101,6cyclictest0-21swapper/319:10:023
558099101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional