You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 15:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 02, 2025 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3001299643,34cyclictest27234-21sh10:00:142
2999699484,5cyclictest131rcu_preempt09:25:160
2999699482,6cyclictest131rcu_preempt12:10:130
2999699482,3cyclictest131rcu_preempt10:20:150
3000099472,41cyclictest17364-21ssh10:15:141
30020994522,14cyclictest25366irq/58-eth010:00:133
3000099443,36cyclictest4120-21ssh12:15:161
3002099434,21cyclictest25366irq/58-eth010:35:133
3002099418,23cyclictest25366irq/58-eth012:10:133
3001299418,26cyclictest21846irq/40-dwc2_hso08:00:142
3002099404,26cyclictest25366irq/58-eth011:05:143
3001299401,30cyclictest21846irq/40-dwc2_hso08:40:142
3000099402,29cyclictest24863-21ssh10:20:141
3002099384,20cyclictest25366irq/58-eth010:30:133
29996993726,3cyclictest12-21ksoftirqd/009:30:120
3002099353,20cyclictest25366irq/58-eth011:15:143
3002099352,21cyclictest25366irq/58-eth008:50:113
3002099343,20cyclictest25366irq/58-eth011:50:143
30012993312,13cyclictest21126-21apt-get09:35:142
3002099326,19cyclictest25366irq/58-eth009:10:133
3000099323,20cyclictest31995-21ssh10:25:141
3001299305,6cyclictest27-21ksoftirqd/210:20:202
3001299301,17cyclictest21728-21ssh12:25:142
3002099292,18cyclictest25366irq/58-eth011:45:163
3002099292,16cyclictest25366irq/58-eth011:20:133
3001299294,14cyclictest21846irq/40-dwc2_hso11:05:122
3001299293,16cyclictest21846irq/40-dwc2_hso08:20:142
3001299292,20cyclictest0-21swapper/209:10:132
3001299292,20cyclictest0-21swapper/208:50:132
30012992913,12cyclictest14207-21apt-get10:35:132
3001299291,17cyclictest27845-21ssh12:10:152
3001299291,16cyclictest6860-21sh10:30:162
3000099297,14cyclictest3126-21kworker/1:011:55:151
2999699290,23cyclictest0-21swapper/012:15:170
3001299282,19cyclictest0-21swapper/210:05:152
3001299282,15cyclictest21846irq/40-dwc2_hso10:10:132
3001299282,15cyclictest21846irq/40-dwc2_hso10:10:132
2999699289,13cyclictest0-21swapper/011:55:130
3002099274,15cyclictest25366irq/58-eth011:00:143
30020992714,6cyclictest0-21swapper/309:25:153
3001299273,16cyclictest21846irq/40-dwc2_hso08:10:132
3001299272,15cyclictest21846irq/40-dwc2_hso11:25:162
30012992713,9cyclictest13582-21apt-get09:30:132
30012992711,11cyclictest21484-21apt-get10:40:152
30012992711,11cyclictest21484-21apt-get10:40:142
3001299266,9cyclictest27-21ksoftirqd/211:22:132
3001299264,8cyclictest27-21ksoftirqd/211:40:202
3001299264,6cyclictest27-21ksoftirqd/207:55:022
3001299264,6cyclictest27-21ksoftirqd/207:55:012
3001299262,9cyclictest0-21swapper/208:55:152
3001299262,13cyclictest0-21swapper/210:52:372
30012992616,8cyclictest261rcuc/211:48:032
30000992613,7cyclictest0-21swapper/109:30:151
3000099261,16cyclictest2596-21df_inode10:05:151
3002099254,14cyclictest0-21swapper/311:55:143
3002099253,18cyclictest25366irq/58-eth009:50:113
3001299254,13cyclictest27-21ksoftirqd/212:07:542
3001299253,4cyclictest8457-21apt-get11:15:032
3001299252,9cyclictest0-21swapper/207:35:152
3001299252,16cyclictest21846irq/40-dwc2_hso12:30:152
3001299252,14cyclictest21846irq/40-dwc2_hso12:00:132
3001299251,4cyclictest271ksoftirqd/212:39:152
3001299251,16cyclictest8613-21sed11:35:222
2999699252,12cyclictest32002-21ssh10:25:130
3001299247,8cyclictest27-21ksoftirqd/211:52:172
3001299247,15cyclictest261rcuc/210:15:312
3001299243,13cyclictest21846irq/40-dwc2_hso07:25:132
3001299242,4cyclictest271ksoftirqd/212:24:342
3001299242,15cyclictest21846irq/40-dwc2_hso07:30:142
30012992414,8cyclictest261rcuc/210:29:032
30012992414,4cyclictest2663-21apt-get07:15:152
3001299241,16cyclictest0-21swapper/207:40:162
30000992418,5cyclictest0-21swapper/111:40:171
2999699242,17cyclictest14224-21ssh10:35:150
3001299237,11cyclictest27-21ksoftirqd/211:31:042
3001299237,10cyclictest27-21ksoftirqd/212:15:152
3001299235,4cyclictest27-21ksoftirqd/209:17:322
3001299232,8cyclictest27-21ksoftirqd/209:56:262
3001299232,13cyclictest21846irq/40-dwc2_hso08:15:142
30012992314,7cyclictest261rcuc/209:52:542
30012992314,3cyclictest261rcuc/209:40:022
30012992314,3cyclictest261rcuc/209:40:012
3001299231,13cyclictest0-21swapper/211:00:492
30000992314,7cyclictest26578-21sshd12:25:341
3000099231,18cyclictest7440-21find07:20:131
3000099231,17cyclictest30966-21ssh09:20:151
2999699232,13cyclictest4189-21ssh09:45:170
29996992312,6cyclictest0-21swapper/010:10:030
3002099223,11cyclictest25366irq/58-eth012:35:143
3002099221,3cyclictest0-21swapper/311:40:173
3001299229,10cyclictest10428-21kthreadcore07:55:222
3001299226,7cyclictest27-21ksoftirqd/209:25:542
3001299222,4cyclictest0-21swapper/210:45:542
3001299222,4cyclictest0-21swapper/210:45:532
3001299221,6cyclictest2764-21ls07:45:272
3001299221,15cyclictest1419-21cut11:10:202
3001299221,15cyclictest1419-21cut11:10:192
3001299221,12cyclictest18384-21fschecks_count08:05:162
3001299220,19cyclictest6223-21cpuspeed09:00:152
3001299220,13cyclictest27-21ksoftirqd/209:20:042
30000992213,5cyclictest0-21swapper/107:15:021
3000099221,16cyclictest21508-21ssh10:40:161
3000099221,16cyclictest21508-21ssh10:40:161
2999699228,3cyclictest131rcu_preempt09:55:170
3002099216,6cyclictest0-21swapper/311:25:153
3002099213,13cyclictest0-21swapper/311:10:303
3002099213,13cyclictest0-21swapper/311:10:293
3002099212,12cyclictest0-21swapper/312:05:123
3001299216,7cyclictest261rcuc/207:10:172
3001299215,13cyclictest16273-21awk10:55:412
3001299211,11cyclictest10043-21wc08:30:182
3001299211,10cyclictest6001-21cut09:45:232
30012992110,8cyclictest4949-21apt-get11:55:142
3000099213,4cyclictest131rcu_preempt10:50:241
3000099210,15cyclictest27830-21ssh12:10:141
29996992115,3cyclictest0-21swapper/010:30:160
29996992113,6cyclictest21842-21sshd11:21:410
2999699211,17cyclictest21507-21sh10:40:160
2999699211,17cyclictest21507-21sh10:40:150
2999699211,17cyclictest14247-21ssh11:40:170
3002099203,12cyclictest0-21swapper/310:25:143
3002099203,11cyclictest0-21swapper/307:15:043
3002099202,11cyclictest0-21swapper/310:15:133
3002099202,11cyclictest0-21swapper/308:05:143
30020992011,7cyclictest0-21swapper/312:34:513
3002099201,13cyclictest0-21swapper/308:35:293
30020992010,7cyclictest25366irq/58-eth012:25:153
3001299208,9cyclictest5146-21fschecks_count08:25:162
3001299201,9cyclictest30121-21latency_hist07:10:012
3001299201,16cyclictest24213-21cpu08:45:162
3001299201,10cyclictest16676-21kthreadcore08:35:212
3001299201,10cyclictest15472-21cron09:10:012
3000099207,9cyclictest21-21ksoftirqd/109:05:131
3000099201,4cyclictest4875-21irqstats09:45:221
30000992012,5cyclictest0-21swapper/110:35:141
2999699203,12cyclictest0-21swapper/009:00:130
2999699203,11cyclictest0-21swapper/009:05:140
2999699202,15cyclictest21704-21apt-get12:25:140
2999699201,6cyclictest12838-21sed12:00:170
2999699201,15cyclictest26642-21apt-get07:40:150
2999699201,15cyclictest21131-21ssh09:35:140
2999699201,15cyclictest12894-21ssh12:20:140
29996992010,7cyclictest32593-21ps10:45:300
29996992010,7cyclictest32593-21ps10:45:290
3002099194,11cyclictest0-21swapper/310:40:163
3002099194,11cyclictest0-21swapper/310:40:153
3002099192,11cyclictest0-21swapper/311:35:143
3002099192,11cyclictest0-21swapper/307:55:163
3002099191,13cyclictest0-21swapper/310:50:153
3002099191,13cyclictest0-21swapper/307:10:153
3002099191,12cyclictest0-21swapper/308:25:023
3002099191,11cyclictest0-21swapper/308:30:183
3002099191,10cyclictest0-21swapper/309:10:013
30020991910,7cyclictest0-21swapper/309:57:583
3001299191,9cyclictest10784-21grep07:20:232
3000099196,3cyclictest15479-21diskmemload09:16:461
3000099194,3cyclictest131rcu_preempt11:35:251
3000099193,4cyclictest21-21ksoftirqd/110:30:241
3000099193,4cyclictest21-21ksoftirqd/109:11:161
3000099193,11cyclictest6131-21ssh09:25:141
3000099192,9cyclictest0-21swapper/112:09:561
3000099192,13cyclictest746-21sendmail-mta11:10:171
3000099192,13cyclictest746-21sendmail-mta11:10:171
30000991911,6cyclictest1-21systemd11:05:181
30000991911,6cyclictest1-21systemd07:39:321
3000099191,15cyclictest9940-21ssh10:10:141
3000099191,15cyclictest9940-21ssh10:10:131
3000099191,13cyclictest23058-21apt-get08:10:141
3000099191,12cyclictest18065-21ntp_states08:35:271
30000991910,6cyclictest9879-21kthreadcore11:15:211
30000991910,3cyclictest15479-21diskmemload11:53:541
30000991910,3cyclictest15479-21diskmemload09:43:501
30000991910,3cyclictest15479-21diskmemload09:43:491
3000099190,15cyclictest13161-21ls12:20:181
2999699193,11cyclictest0-21swapper/010:15:130
2999699192,13cyclictest26088-21ssh11:05:140
29996991913,3cyclictest12-21ksoftirqd/011:25:150
29996991910,7cyclictest23348-21sh09:15:170
3002099189,4cyclictest0-21swapper/310:55:213
3002099188,3cyclictest0-21swapper/312:20:163
3002099185,9cyclictest0-21swapper/309:30:153
3002099182,9cyclictest0-21swapper/307:20:143
3002099182,10cyclictest0-21swapper/309:40:143
3002099182,10cyclictest0-21swapper/309:40:133
3002099182,10cyclictest0-21swapper/308:45:173
3002099182,10cyclictest0-21swapper/308:40:133
3002099182,10cyclictest0-21swapper/308:25:133
3002099182,10cyclictest0-21swapper/308:10:153
3002099182,10cyclictest0-21swapper/307:40:173
3002099181,13cyclictest0-21swapper/307:30:293
3002099181,12cyclictest0-21swapper/311:30:283
30020991810,5cyclictest25366irq/58-eth009:15:153
3000099189,3cyclictest15479-21diskmemload12:00:431
3000099188,8cyclictest31639-21fw_packets07:45:151
3000099182,2cyclictest131rcu_preempt10:59:081
3000099181,5cyclictest24708-21irqrtprio08:45:211
3000099181,3cyclictest131rcu_preempt12:38:461
3000099181,3cyclictest131rcu_preempt12:30:051
30000991810,6cyclictest25422-21sshd09:57:411
30000991810,6cyclictest21569-21sshd07:31:411
2999699189,7cyclictest11899-21perl09:50:160
2999699188,7cyclictest30294-21df07:10:140
2999699182,13cyclictest22023-21kthreadcore12:05:200
2999699182,11cyclictest28869-21/usr/sbin/munin09:40:330
2999699182,11cyclictest28869-21/usr/sbin/munin09:40:320
2999699181,9cyclictest26435-21/usr/sbin/munin11:45:330
29996991812,4cyclictest0-21swapper/007:20:310
29996991811,5cyclictest12-21ksoftirqd/008:50:130
2999699181,12cyclictest0-21swapper/007:50:140
2999699181,12cyclictest0-21swapper/007:50:130
3002099179,6cyclictest0-21swapper/312:00:293
3002099179,6cyclictest0-21swapper/309:20:583
3002099177,5cyclictest0-21swapper/310:10:053
3002099177,5cyclictest0-21swapper/310:10:053
3002099172,11cyclictest25366irq/58-eth009:45:173
3002099172,10cyclictest0-21swapper/309:35:143
3002099172,10cyclictest0-21swapper/308:00:133
3002099172,10cyclictest0-21swapper/307:50:143
3002099172,10cyclictest0-21swapper/307:50:133
3002099172,10cyclictest0-21swapper/307:45:123
30020991711,4cyclictest0-21swapper/310:20:063
3000099179,6cyclictest30705-21memory10:00:241
3000099178,7cyclictest25739-21cat09:35:311
3000099177,8cyclictest20912-21cut08:05:201
3000099173,7cyclictest17938-21latency_hist08:05:021
3000099173,11cyclictest29881-21kthreadcore10:45:221
3000099173,11cyclictest29881-21kthreadcore10:45:221
3000099173,10cyclictest0-21swapper/109:00:141
3000099172,9cyclictest0-21swapper/107:50:141
3000099172,9cyclictest0-21swapper/107:50:141
3000099172,3cyclictest131rcu_preempt09:00:011
3000099172,11cyclictest32673-21apt-get08:20:151
3000099172,10cyclictest11698-21ssh09:50:141
3000099172,10cyclictest0-21swapper/111:00:151
3000099171,8cyclictest11638-21kthreadcore08:30:211
3000099171,4cyclictest52846usb-storage08:15:171
3000099171,14cyclictest29017-21apt-get08:50:121
30000991711,4cyclictest12411-21fschecks_time07:25:151
30000991710,5cyclictest23937-21grep11:45:241
30000991710,4cyclictest31624-21irqcore11:30:181
30000991710,4cyclictest31341-21latency_hist07:45:021
3000099170,3cyclictest131rcu_preempt11:22:311
2999699179,6cyclictest3759-21ssh11:30:440
2999699179,6cyclictest3702-21sh08:20:280
2999699177,7cyclictest19577-21kthreadcore07:30:220
2999699174,11cyclictest15993-21timerwakeupswit07:25:280
2999699173,11cyclictest6965-21ssh12:35:150
2999699172,10cyclictest0-21swapper/011:00:160
29996991711,4cyclictest0-21swapper/008:05:010
29996991711,3cyclictest0-21swapper/010:55:140
2999699171,13cyclictest0-21swapper/008:30:160
2999699171,12cyclictest9839-21ssh10:10:050
2999699171,12cyclictest9839-21ssh10:10:040
29996991710,5cyclictest27651-21ntp_kernel_pll_08:45:280
29996991710,4cyclictest1484-21cut08:55:170
3002099168,6cyclictest0-21swapper/312:15:083
3002099168,6cyclictest0-21swapper/310:48:213
3002099168,6cyclictest0-21swapper/310:48:203
3002099168,5cyclictest25366irq/58-eth010:05:143
3000099169,4cyclictest0-21swapper/108:40:171
3000099168,6cyclictest8496-21sh08:25:261
3000099168,6cyclictest26937-21kthreadcore11:25:251
3000099163,11cyclictest21-21ksoftirqd/107:15:231
30000991610,4cyclictest0-21swapper/107:55:331
2999699168,5cyclictest0-21swapper/007:45:170
2999699166,7cyclictest27603-21tail10:00:180
2999699163,10cyclictest4216-21kthreadcore07:15:230
2999699163,10cyclictest16470-21kthreadcore08:35:210
2999699162,11cyclictest31063-21ssh09:20:150
2999699162,11cyclictest1215-21/usr/sbin/munin11:10:210
2999699162,11cyclictest1215-21/usr/sbin/munin11:10:200
2999699161,8cyclictest4205-21/usr/sbin/munin10:50:150
29996991611,3cyclictest10126-21sshd11:35:240
29996991611,3cyclictest0-21swapper/009:10:120
2999699161,12cyclictest5205-21fschecks_time08:25:160
2999699161,12cyclictest31263-21ntp_192.168.11408:15:260
29996991610,3cyclictest29811-21sh11:50:150
3002099156,5cyclictest0-21swapper/307:35:143
3002099152,9cyclictest0-21swapper/307:25:133
2999699153,10cyclictest25650-21grep07:35:300
2999699153,10cyclictest23450-21sed08:10:180
2999699151,11cyclictest9218-21kernelversion07:55:200
2999699151,11cyclictest30560-21ssh12:30:150
2999699151,11cyclictest19402-21apt-get08:40:130
2999699151,11cyclictest17937-21apt-get08:05:050
2999699151,10cyclictest8876-21/usr/sbin/munin11:15:130
3002099142,5cyclictest0-21swapper/308:20:013
3000099141,10cyclictest30007-21latency_hist07:10:011
2999699141,10cyclictest30122-21latency_hist07:10:020
3002099138,3cyclictest0-21swapper/309:00:173
3002099133,8cyclictest0-21swapper/309:00:013
3002099111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional