You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-31 - 12:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 31, 2025 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3224699662,48cyclictest10459-21ssh22:40:140
3224699601,48cyclictest3173-21ssh21:20:130
3224699583,45cyclictest14502-21ssh00:15:150
3225499563,12cyclictest131rcu_preempt21:05:131
3224699472,4cyclictest8462-21ssh00:30:170
32275994533,6cyclictest28366irq/58-eth023:35:143
3225499449,28cyclictest21-21ksoftirqd/100:20:151
32267994311,13cyclictest27-21ksoftirqd/223:05:152
32275994228,9cyclictest33-21ksoftirqd/319:30:143
3226799421,26cyclictest3162-21ssh21:20:132
32275994134,4cyclictest28366irq/58-eth000:10:143
32267994114,17cyclictest27-21ksoftirqd/223:00:142
3226799411,18cyclictest18638-21ssh21:10:122
3225499411,28cyclictest21848-21ssh21:50:131
3226799402,26cyclictest8442-21ssh22:20:132
3225499402,30cyclictest668-21ssh23:30:121
3225499398,24cyclictest21-21ksoftirqd/119:15:121
32254993828,7cyclictest21-21ksoftirqd/123:55:011
32254993820,9cyclictest21-21ksoftirqd/121:45:151
3227599372,18cyclictest28366irq/58-eth019:55:153
3226799374,23cyclictest9355-21diskmemload23:35:142
3225499373,25cyclictest20266-21diskmemload21:30:141
3225499372,28cyclictest21839-21ssh23:05:161
32275993624,8cyclictest28366irq/58-eth023:25:133
3224699361,9cyclictest572-21uname19:10:190
3224699361,10cyclictest2727-21kthreadcore19:45:210
3224699361,10cyclictest2727-21kthreadcore19:45:210
3226799358,20cyclictest18245-21diskmemload21:16:552
3226799358,20cyclictest18245-21diskmemload21:16:552
32267993525,6cyclictest27-21ksoftirqd/222:50:142
32267993519,11cyclictest0-21swapper/220:15:142
3225499353,8cyclictest131rcu_preempt19:30:131
3224699352,5cyclictest0-21swapper/023:23:420
3224699352,5cyclictest0-21swapper/023:23:410
3224699352,29cyclictest16983-21ssh22:25:130
3224699351,9cyclictest17028-21unixbench_multi23:00:290
3224699351,9cyclictest0-21swapper/021:45:200
3224699351,8cyclictest24560-21sed19:35:180
3224699351,10cyclictest25925-21grep21:50:300
3225499344,18cyclictest21-21ksoftirqd/120:05:131
32254993419,10cyclictest21-21ksoftirqd/119:10:131
3225499341,4cyclictest131rcu_preempt19:50:131
3224699341,9cyclictest24382-21kthreadcore20:40:210
3224699341,9cyclictest19845-21cut23:40:200
3224699341,9cyclictest15887-21cut19:25:190
3224699341,9cyclictest13968-21kthreadcore23:55:220
3224699341,8cyclictest2628-21sed22:35:170
3224699341,11cyclictest2801-21tr23:30:220
3224699341,10cyclictest9125-21kthreadcore23:15:210
3224699341,10cyclictest5511-21kthreadcore20:55:210
3224699341,10cyclictest30855-21kthreadcore21:35:220
3224699341,10cyclictest28854-21cut20:45:230
3224699341,10cyclictest23081-21kthreadcore23:05:220
3224699341,10cyclictest17844-21kthreadcore20:35:200
3224699340,8cyclictest31440-21ntp_states22:50:270
3224699337,3cyclictest0-21swapper/022:15:280
3224699334,20cyclictest26403-21kworker/u8:200:10:120
3224699331,9cyclictest8733-21usage_Anybus19:15:260
3224699331,9cyclictest25317-21kthreadcore00:20:230
3224699331,9cyclictest23958-21/usr/sbin/munin21:30:290
3224699331,9cyclictest21080-21switchtime00:35:280
3224699331,9cyclictest20909-21irqcore20:05:180
3224699331,9cyclictest10148-21ntp_offset22:00:270
3224699331,8cyclictest3806-21kthreadcore20:20:190
3224699331,8cyclictest30766-21ntpq21:15:240
3224699331,8cyclictest30766-21ntpq21:15:230
3224699331,8cyclictest29181-21ls20:10:270
3224699331,8cyclictest1993-21grep21:55:310
3224699331,8cyclictest11108-21fschecks_count19:55:180
3224699331,7cyclictest8408-21grep21:40:260
3224699331,7cyclictest22826-21wc19:30:260
3224699331,7cyclictest11370-21cut23:35:220
3227599322,15cyclictest28366irq/58-eth020:00:133
3226799326,14cyclictest22346irq/40-dwc2_hso23:20:122
3226799326,14cyclictest22346irq/40-dwc2_hso23:20:112
3226799323,7cyclictest0-21swapper/222:32:082
32267993214,8cyclictest0-21swapper/221:05:142
32267993214,12cyclictest24371-21ssh23:25:132
3224699325,5cyclictest23210-21cut00:00:230
3224699322,10cyclictest10335-21ssh22:57:140
3224699321,8cyclictest25694-21ssh22:47:520
3224699321,8cyclictest15444-21kthreadcore20:30:230
3224699321,8cyclictest15444-21kthreadcore20:30:220
3224699321,8cyclictest15312-21ntp_192.168.11421:25:250
3224699321,8cyclictest1113-21idleruntime-cro19:45:010
3224699321,7cyclictest29799-21sed00:05:200
3224699321,7cyclictest27918-21kthreadcore23:45:220
3224699321,7cyclictest17732-21usage_Anybus21:05:290
3224699321,10cyclictest18859-21ntp_states22:05:270
3224699321,10cyclictest18859-21ntp_states22:05:260
3224699321,10cyclictest1845-21kthreadcore20:50:220
32246993210,19cyclictest18757-21entropy21:10:160
3224699320,8cyclictest5867-21kthreadcore23:50:230
3226799313,21cyclictest0-21swapper/220:21:412
3226799312,18cyclictest22346irq/40-dwc2_hso22:40:152
3226799312,15cyclictest27-21ksoftirqd/200:20:022
3225499312,21cyclictest31589-21apt-get20:50:151
32254993118,6cyclictest18245-21diskmemload21:25:141
3224699311,8cyclictest31985-21kthreadcore20:15:200
3224699311,8cyclictest12008-21ntp_192.168.11422:20:240
3224699311,7cyclictest9101-21/usr/sbin/munin21:00:160
3224699311,7cyclictest7647-21fschecks_count20:25:150
3224699311,7cyclictest27708-21/usr/sbin/munin22:10:330
3224699311,7cyclictest25621-21/usr/sbin/munin22:30:120
3224699311,10cyclictest1496-21tr00:25:220
3226799303,20cyclictest0-21swapper/219:40:172
3226799302,21cyclictest16561-21ssh00:34:422
32267993016,9cyclictest0-21swapper/223:50:152
32267993011,16cyclictest27-21ksoftirqd/222:25:132
3224699302,5cyclictest30690-21df_inode23:10:130
3224699301,7cyclictest24375-21ssh23:25:140
3224699301,6cyclictest9750-21/usr/sbin/munin19:20:160
3227599291,17cyclictest28366irq/58-eth021:15:123
3227599291,17cyclictest28366irq/58-eth021:15:113
3226799293,15cyclictest22346irq/40-dwc2_hso19:10:142
3225499292,17cyclictest1930-21ssh22:35:121
3224699291,8cyclictest9797-21/usr/sbin/munin19:50:280
3224699291,10cyclictest17966-21kthreadcore20:00:210
3227599283,16cyclictest0-21swapper/322:17:403
32275992819,7cyclictest0-21swapper/323:55:153
32275992816,9cyclictest28366irq/58-eth023:45:153
3226799282,20cyclictest0-21swapper/221:01:402
3226799282,19cyclictest0-21swapper/219:25:402
3226799282,15cyclictest23657-21ssh22:10:162
32254992812,13cyclictest8651-21ssh00:30:201
3226799278,13cyclictest27-21ksoftirqd/221:54:082
3226799275,8cyclictest27-21ksoftirqd/221:48:512
3226799274,16cyclictest22346irq/40-dwc2_hso20:45:162
3226799272,19cyclictest30591-21ssh23:10:122
3226799272,14cyclictest4156-21sh22:55:142
3226799272,11cyclictest27-21ksoftirqd/220:55:142
3226799272,10cyclictest0-21swapper/219:20:152
3226799271,19cyclictest10705-21ls23:15:282
3225499272,20cyclictest4791-21ssh21:40:131
3225499272,19cyclictest3856-21sh23:11:411
3225499272,19cyclictest18644-21ssh21:10:151
3227599263,11cyclictest0-21swapper/322:50:403
3227599262,21cyclictest28366irq/58-eth000:00:153
32275992619,4cyclictest0-21swapper/300:30:173
3226799268,7cyclictest27-21ksoftirqd/200:00:502
3226799264,13cyclictest22346irq/40-dwc2_hso20:50:142
32267992616,3cyclictest261rcuc/222:47:252
32267992614,8cyclictest26712-21apt-get23:45:142
32267992613,5cyclictest0-21swapper/219:55:152
32267992612,9cyclictest17182-21/usr/sbin/munin00:35:192
3226799261,17cyclictest29630-21ls00:05:192
3227599259,10cyclictest28366irq/58-eth023:10:133
3227599256,11cyclictest28366irq/58-eth000:25:133
32275992515,6cyclictest33-21ksoftirqd/322:00:153
3226799258,10cyclictest656-21apt-get23:30:122
3226799257,12cyclictest27-21ksoftirqd/222:36:352
3226799256,11cyclictest22346irq/40-dwc2_hso19:50:132
3226799254,15cyclictest24335-21timerwakeupswit20:05:282
3226799253,15cyclictest22346irq/40-dwc2_hso20:40:152
32267992515,4cyclictest261rcuc/221:43:162
3226799251,11cyclictest5725-21ssh00:10:132
3227599242,10cyclictest0-21swapper/321:21:413
3226799246,8cyclictest27-21ksoftirqd/200:23:502
3226799246,4cyclictest27-21ksoftirqd/221:55:052
3226799245,7cyclictest27-21ksoftirqd/222:04:252
3226799242,4cyclictest32230-21apt-get22:15:152
32267992419,3cyclictest261rcuc/223:57:242
32267992419,3cyclictest261rcuc/200:26:352
32267992415,3cyclictest261rcuc/221:25:382
3225499249,11cyclictest12035-21kworker/1:020:30:151
3225499249,11cyclictest12035-21kworker/1:020:30:141
3225499248,12cyclictest21-21ksoftirqd/123:24:421
3225499248,12cyclictest21-21ksoftirqd/123:24:421
3225499243,12cyclictest633-21Async12:11:531
3225499243,12cyclictest633-21Async12:11:531
3225499241,18cyclictest29257-21ssh22:30:241
32254992410,8cyclictest0-21swapper/119:40:151
3227599233,17cyclictest0-21swapper/323:22:213
3227599233,17cyclictest0-21swapper/323:22:203
3227599232,14cyclictest28466irq/59-eth022:35:123
3227599232,13cyclictest0-21swapper/322:05:143
3227599232,13cyclictest0-21swapper/322:05:133
3226799237,8cyclictest27-21ksoftirqd/222:06:182
3226799237,8cyclictest27-21ksoftirqd/222:06:182
3226799236,9cyclictest0-21swapper/219:15:122
3226799236,13cyclictest0-21swapper/220:25:122
3226799235,12cyclictest27-21ksoftirqd/221:30:062
3226799231,6cyclictest27-21ksoftirqd/220:10:122
3226799231,19cyclictest0-21swapper/220:35:142
32267992310,4cyclictest27-21ksoftirqd/223:43:232
32254992313,5cyclictest0-21swapper/122:05:151
32254992313,5cyclictest0-21swapper/122:05:141
32254992312,7cyclictest4395-21cron21:40:011
3227599223,13cyclictest0-21swapper/322:20:413
3227599222,13cyclictest28366irq/58-eth023:00:123
32275992213,5cyclictest0-21swapper/322:55:233
32275992211,8cyclictest0-21swapper/321:50:183
32267992213,7cyclictest0-21swapper/221:35:132
32254992214,5cyclictest16014-21grep20:00:171
32254992213,6cyclictest22829-21sendmail00:20:001
3225499221,11cyclictest27420-21latency_hist22:50:021
3227599215,14cyclictest28366irq/58-eth021:35:123
3227599211,14cyclictest0-21swapper/321:00:013
3227599211,13cyclictest0-21swapper/323:05:163
3225499212,14cyclictest23801-21ssh00:37:421
32254992112,6cyclictest9314-21ssh00:10:251
32254992110,8cyclictest32321-21cpuspeed22:15:161
3227599209,6cyclictest0-21swapper/323:30:583
3227599202,13cyclictest0-21swapper/321:11:423
32275992012,5cyclictest0-21swapper/323:42:213
3227599201,12cyclictest28366irq/58-eth021:40:133
3226799205,3cyclictest27-21ksoftirqd/219:30:232
3226799203,11cyclictest22346irq/40-dwc2_hso19:35:122
3226799201,9cyclictest2915-21kthreadcore19:45:212
3226799201,9cyclictest2915-21kthreadcore19:45:202
3226799201,9cyclictest13378-21kthreadcore20:30:212
3226799201,9cyclictest13378-21kthreadcore20:30:212
3226799201,10cyclictest20404-21grep20:05:012
32254992013,4cyclictest14101-21sh22:40:261
32254992013,4cyclictest0-21swapper/119:55:301
32254992012,6cyclictest1370-21ssh00:25:221
32254992012,4cyclictest28192-21mailstats23:25:291
32254992011,6cyclictest7959-21ssh21:20:311
32254992011,6cyclictest18111-21ssh23:40:151
3227599199,8cyclictest0-21swapper/300:24:253
3227599195,11cyclictest0-21swapper/321:55:163
3227599193,11cyclictest0-21swapper/323:16:413
3227599193,11cyclictest0-21swapper/322:31:413
3227599193,11cyclictest0-21swapper/321:27:423
3227599193,11cyclictest0-21swapper/320:20:413
3227599193,11cyclictest0-21swapper/320:15:403
3227599193,11cyclictest0-21swapper/300:37:413
3227599193,10cyclictest0-21swapper/321:00:413
3227599192,11cyclictest0-21swapper/321:05:133
3227599192,10cyclictest0-21swapper/319:15:053
3227599191,13cyclictest0-21swapper/320:45:303
3227599191,12cyclictest0-21swapper/300:10:023
3225499199,7cyclictest9139-21cut23:15:201
3225499197,9cyclictest8557-21cron21:00:011
3225499193,9cyclictest0-21swapper/122:20:401
3225499193,11cyclictest11810-21apt-get23:55:151
3225499191,9cyclictest6723-21hwlatdetect22:00:171
32254991911,5cyclictest4878-21kernelversion22:55:211
3225499191,15cyclictest30394-21ssh21:55:161
32254991911,5cyclictest26689-21/usr/sbin/munin23:45:301
3225499191,13cyclictest9316-21/usr/sbin/munin23:35:341
3225499191,10cyclictest27630-21kthreadcore20:45:211
32254991910,6cyclictest31822-21cron22:15:011
3227599189,6cyclictest28366irq/58-eth023:50:143
3227599183,8cyclictest0-21swapper/321:30:163
3227599183,10cyclictest0-21swapper/319:25:413
3227599182,6cyclictest0-21swapper/322:10:173
3227599182,11cyclictest0-21swapper/322:29:413
3227599182,11cyclictest0-21swapper/321:48:423
3227599182,11cyclictest0-21swapper/319:45:153
3227599182,11cyclictest0-21swapper/319:45:143
3227599182,10cyclictest0-21swapper/320:30:043
3227599182,10cyclictest0-21swapper/320:30:043
3227599182,10cyclictest0-21swapper/320:05:133
3227599182,10cyclictest0-21swapper/319:35:143
32275991811,4cyclictest0-21swapper/322:45:293
32275991810,5cyclictest28366irq/58-eth000:15:153
3225499186,9cyclictest12207-21latency_hist20:30:011
3225499185,9cyclictest13170-21grep23:00:171
3225499183,10cyclictest0-21swapper/120:15:411
3225499182,13cyclictest17085-21ssh22:25:151
3225499181,9cyclictest24107-21/usr/sbin/munin19:35:271
3225499181,14cyclictest3331-21kthreadcore20:20:181
3225499181,11cyclictest27790-21ssh22:50:141
32254991810,5cyclictest18245-21diskmemload00:00:251
3227599179,6cyclictest0-21swapper/322:40:493
3227599172,10cyclictest0-21swapper/319:50:153
3226799171,13cyclictest32287-21latency_hist19:10:022
3225499179,6cyclictest582-21sh00:05:281
3225499179,6cyclictest0-21swapper/121:00:121
3225499178,6cyclictest15091-21cut19:25:181
3225499178,6cyclictest14275-21sh19:25:001
3225499173,10cyclictest0-21swapper/120:36:401
3225499171,8cyclictest21983-21/usr/sbin/munin20:40:271
3225499171,3cyclictest131rcu_preempt19:45:281
3225499171,3cyclictest131rcu_preempt19:45:271
32254991711,4cyclictest28833-21nfs4_client20:10:231
3227599162,10cyclictest0-21swapper/320:35:143
3227599143,8cyclictest0-21swapper/320:30:023
3227599142,9cyclictest0-21swapper/319:40:173
3227599141,7cyclictest0-21swapper/319:20:143
3225499141,9cyclictest32358-21latency_hist19:10:021
3224699141,10cyclictest32298-21latency_hist19:10:010
3227599133,8cyclictest0-21swapper/320:10:273
3227599131,7cyclictest28366irq/58-eth020:40:153
3227599131,5cyclictest0-21swapper/320:50:203
3227599122,4cyclictest0-21swapper/319:10:143
3227599103,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional