You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 02:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Mar 30, 2026 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1609924530,27sleep37950irq/56-dwc_otg_19:05:323
1657322900,2sleep20-21swapper/219:05:402
208402920,2sleep220843-21latency_hist21:40:122
17346998919,47cyclictest26441-21fschecks_time22:25:290
17346998620,46cyclictest22864-21aten_rbpower_en19:25:270
17346998414,46cyclictest29457-21ps21:10:370
17346998314,51cyclictest6089-21proc_pri22:00:360
250322790,2sleep21734899cyclictest21:00:302
17346997814,50cyclictest20758-21munin-plugin-st19:20:120
17346997712,48cyclictest5559-21/usr/sbin/munin20:05:320
17346997617,39cyclictest26412-21latency_hist21:05:120
17346997517,44cyclictest18674-21grep20:40:390
17346997516,44cyclictest31610-21ssh21:52:140
17346997513,41cyclictest31984-21fschecks_count23:50:280
17346997512,46cyclictest15429-21modprobe00:10:110
17346997414,46cyclictest15705-21/usr/sbin/munin23:30:320
17346997414,46cyclictest15705-21/usr/sbin/munin23:30:320
1734699741,39cyclictest0-21swapper/022:35:410
17346997413,45cyclictest17555-21ssh21:35:310
1734699738,57cyclictest0-21swapper/021:43:510
1734699738,48cyclictest31945-21sed23:10:380
17346997317,40cyclictest5853-21ssh23:19:120
17346997313,43cyclictest20303-21switchtime23:35:380
17346997222,18cyclictest61-21kswapd023:20:250
17346997222,18cyclictest61-21kswapd023:20:250
1734699721,59cyclictest0-21swapper/023:00:300
17346997115,44cyclictest20381-21rm22:56:360
17346997114,35cyclictest26970-21ssh23:44:450
17346997113,44cyclictest2930-21cat20:00:120
17346997016,43cyclictest6607-21sh00:38:040
17346997016,40cyclictest2151-21ssh00:32:290
163402700,1sleep30-21swapper/323:30:403
163402700,1sleep30-21swapper/323:30:403
17346996910,48cyclictest2041-21rm21:16:120
265112680,2sleep30-21swapper/323:05:113
1734699689,38cyclictest0-21swapper/021:05:290
1734699687,47cyclictest22132-21grep19:20:390
1734699687,47cyclictest22132-21grep19:20:390
17346996816,41cyclictest13505-21ssh23:28:080
1734699681,55cyclictest7881-21grep20:10:410
17346996814,25cyclictest29673-21dhclient-script19:44:040
17346996812,45cyclictest18998-21latency_hist19:15:120
1734699671,63cyclictest11452-21tail20:20:390
1734699671,63cyclictest11452-21tail20:20:390
17346996713,39cyclictest29975-21diskmemload23:47:410
17346996713,38cyclictest27102-21fschecks_time23:05:280
17346996710,48cyclictest31654-21ssh00:29:410
106082670,7sleep31734999cyclictest21:26:443
17346996611,46cyclictest22633-21kworker/u8:200:00:310
17346996611,11cyclictest0-21swapper/022:30:410
17346996610,43cyclictest28645-21ssh21:49:240
171262660,8sleep1221ktimersoftd/119:10:121
17346996525,37cyclictest14386-21apt-get22:50:230
17346996514,40cyclictest11500-21unixbench_multi22:45:410
1734699651,13cyclictest24530-21latency_hist21:00:120
1734699649,32cyclictest0-21swapper/021:30:120
1734699648,42cyclictest9598-21cpu22:05:260
1734699647,11cyclictest8396-21runrttasks22:42:260
17346996416,39cyclictest17396-21ssh00:11:130
17346996416,35cyclictest12816-21ssh21:30:210
17346996415,39cyclictest24419-21ssh22:22:460
17346996412,36cyclictest24493-21ssh00:20:290
17346996412,36cyclictest24493-21ssh00:20:290
1734699633,35cyclictest1088-21rm21:55:210
17346996217,41cyclictest22812-21ssh00:18:350
1734699614,52cyclictest0-21swapper/019:50:280
17346996114,35cyclictest15616-21ssh22:11:590
17346996111,38cyclictest4015-21ssh23:55:330
1734699609,31cyclictest0-21swapper/021:21:450
17346996016,32cyclictest3423-21grep20:00:290
17346996013,36cyclictest13848-21latency_hist20:30:120
17346996013,36cyclictest13848-21latency_hist20:30:120
113412600,2sleep20-21swapper/221:28:042
1734699588,41cyclictest0-21swapper/020:35:390
17346995838,5cyclictest21480-21fschecks_time20:50:290
17346995811,31cyclictest21452-21ssh22:19:550
17349995712,10cyclictest1140-21ssh21:55:253
147442570,6sleep316204-21kworker/3:123:30:123
17346995634,11cyclictest27065-21ntpq19:35:330
17346995628,8cyclictest20163-21gzip20:45:360
17347995518,15cyclictest11057-21sh21:27:331
17347995513,31cyclictest0-21swapper/123:20:291
17347995513,31cyclictest0-21swapper/123:20:291
17346995519,6cyclictest31188-21head19:45:400
17346995513,11cyclictest15309-21taskset20:32:510
122302530,2sleep112231-21/usr/sbin/munin00:05:331
17346995210,28cyclictest26012-21chrt19:33:260
17346995110,30cyclictest0-21swapper/020:16:210
1734899505,29cyclictest0-21swapper/219:10:142
17347995015,11cyclictest1119-21apt-get21:55:251
167932503,12sleep00-21swapper/019:06:320
17349994930,12cyclictest421-21ssh21:15:263
17347994911,16cyclictest25647-21ssh23:42:181
17349994823,8cyclictest10252-21apt-get22:45:243
1734899487,21cyclictest0-21swapper/221:55:262
17349994716,5cyclictest28246-21diskmemload22:30:333
17347994723,18cyclictest0-21swapper/119:10:141
17347994717,16cyclictest221ktimersoftd/123:58:581
17347994710,11cyclictest16679-21cat20:35:381
1734899467,25cyclictest0-21swapper/220:05:242
1734899467,23cyclictest0-21swapper/200:00:242
1734899457,25cyclictest0-21swapper/219:25:262
1734899457,20cyclictest31-21ksoftirqd/220:35:392
1734899456,25cyclictest0-21swapper/220:45:242
17347994515,16cyclictest6841-21ssh00:38:301
17347994514,8cyclictest7950irq/56-dwc_otg_23:25:221
17347994512,10cyclictest221ktimersoftd/122:28:381
17347994511,16cyclictest26350-21ssh23:05:021
17347994511,11cyclictest2219-21ssh21:16:311
316252440,5sleep11734799cyclictest22:30:511
1734799449,18cyclictest6681-21rm22:00:531
1734799448,11cyclictest18379-21ssh00:13:021
1734799447,11cyclictest32456-21rm23:11:061
1734799447,11cyclictest23375-21ssh22:20:531
1734799446,23cyclictest27296-21ssh23:05:311
1734799445,26cyclictest81032sed00:00:301
17347994429,9cyclictest7850irq/56-dwc_otg_22:15:321
17347994422,10cyclictest7950irq/56-dwc_otg_21:11:061
17347994416,15cyclictest0-21swapper/122:45:231
17347994415,12cyclictest221ktimersoftd/121:35:521
17347994410,14cyclictest13899-21latency_hist20:30:121
17349994320,8cyclictest7850irq/56-dwc_otg_21:47:053
17349994320,15cyclictest32465-21ssh23:11:073
1734899434,30cyclictest7750irq/56-dwc_otg23:20:122
1734799439,7cyclictest26337-21apt-get21:05:141
1734799436,16cyclictest7950irq/56-dwc_otg_19:55:131
17347994311,8cyclictest22805-21apt-get19:25:251
1734899426,25cyclictest0-21swapper/200:10:262
1734899422,18cyclictest7850irq/56-dwc_otg_21:15:402
1734899421,9cyclictest301ktimersoftd/200:35:412
1734799429,21cyclictest0-21swapper/120:30:301
1734799429,14cyclictest22595-21ssh21:41:031
17347994219,8cyclictest221ktimersoftd/121:47:001
17347994213,4cyclictest221ktimersoftd/100:17:251
236202410,2sleep30-21swapper/322:21:193
1734999412,15cyclictest11-21rcu_preempt22:11:393
1734999411,17cyclictest0-21swapper/323:20:173
1734999411,17cyclictest0-21swapper/323:20:173
1734899417,24cyclictest0-21swapper/221:05:182
1734899417,22cyclictest0-21swapper/223:35:252
1734899414,8cyclictest29-21rcuc/221:49:252
17348994111,6cyclictest2090-21processes19:55:342
1734799416,20cyclictest17943-21sh23:33:301
1734799416,20cyclictest17943-21sh23:33:301
1734799416,19cyclictest25778-21chrt00:21:271
1734799416,19cyclictest25778-21chrt00:21:271
17347994124,9cyclictest7750irq/56-dwc_otg20:21:461
17347994124,9cyclictest7750irq/56-dwc_otg20:21:461
17347994117,12cyclictest7950irq/56-dwc_otg_00:27:171
17347994115,11cyclictest84792chrt22:42:351
17347994112,10cyclictest0-21swapper/100:31:451
1734999408,13cyclictest7950irq/56-dwc_otg_00:15:523
1734999403,21cyclictest7950irq/56-dwc_otg_23:59:063
17349994017,10cyclictest7950irq/56-dwc_otg_00:35:233
1734899408,8cyclictest11-21rcu_preempt21:10:372
1734899406,10cyclictest5851-21latency_hist22:40:122
1734899403,18cyclictest7950irq/56-dwc_otg_00:23:032
1734899403,18cyclictest7950irq/56-dwc_otg_00:23:032
17348994023,7cyclictest31-21ksoftirqd/223:22:092
17348994023,7cyclictest31-21ksoftirqd/223:22:092
17348994015,5cyclictest908-21runrttasks00:05:412
17348994013,15cyclictest301ktimersoftd/222:45:242
1734899401,24cyclictest0-21swapper/220:55:242
1734899401,24cyclictest0-21swapper/220:10:262
1734899401,21cyclictest7850irq/56-dwc_otg_21:20:242
1734799409,11cyclictest5962-21sed20:05:381
1734799405,11cyclictest7960-21sh21:24:261
17347994013,4cyclictest221ktimersoftd/121:30:231
17347994013,11cyclictest32112-21/usr/sbin/munin23:50:291
17349993922,8cyclictest7950irq/56-dwc_otg_21:21:463
17349993920,4cyclictest11629-21/usr/sbin/munin00:05:313
1734899392,16cyclictest0-21swapper/221:43:182
1734899391,23cyclictest32283-21ssh00:30:232
17348993910,8cyclictest7950irq/56-dwc_otg_19:20:352
17348993910,8cyclictest7950irq/56-dwc_otg_19:20:352
1734799398,18cyclictest20593-21chrt20:46:121
1734799396,19cyclictest3814-21awk20:00:361
1734799395,3cyclictest0-21swapper/122:07:541
17347993919,5cyclictest18797-21fschecks_count22:55:261
17347993914,16cyclictest4041-21ssh22:37:121
17347993914,16cyclictest31795-21ssh21:52:351
17347993913,16cyclictest5890-21ssh23:19:171
17347993912,12cyclictest221ktimersoftd/122:50:551
17347993911,7cyclictest13711-21munin-run20:30:111
17347993911,13cyclictest21602-21missed_timers19:20:321
17347993911,13cyclictest21602-21missed_timers19:20:321
1734999388,5cyclictest381ktimersoftd/321:41:263
1734999385,20cyclictest7950irq/56-dwc_otg_22:51:243
17349993822,5cyclictest10535-21sh00:03:493
17349993818,11cyclictest7750irq/56-dwc_otg21:13:503
17349993814,15cyclictest7750irq/56-dwc_otg19:30:463
17349993811,15cyclictest0-21swapper/323:53:203
17348993819,8cyclictest7750irq/56-dwc_otg20:00:342
1734899381,30cyclictest8869-21ls20:15:302
1734899381,30cyclictest13954-21cut22:10:292
1734899381,2cyclictest0-21swapper/223:14:532
1734899381,27cyclictest15572-21ls20:35:112
1734899381,25cyclictest26498-21ssh23:05:122
1734899381,19cyclictest0-21swapper/222:00:372
17348993811,13cyclictest301ktimersoftd/222:25:522
1734799389,12cyclictest28246-21diskmemload22:11:421
17347993819,7cyclictest7950irq/56-dwc_otg_23:36:191
17347993810,7cyclictest28812-21unixbench_multi23:45:401
17347993810,17cyclictest31506-21taskset19:49:211
1734999379,14cyclictest6012-21ntp_states22:00:353
1734999378,14cyclictest13528-1kworker/3:0H19:55:183
1734999376,12cyclictest23257-21/usr/sbin/munin19:25:323
1734999374,21cyclictest7950irq/56-dwc_otg_22:06:013
1734999373,18cyclictest0-21swapper/322:36:543
17349993721,7cyclictest32395-21ssh21:53:423
17349993715,9cyclictest7850irq/56-dwc_otg_00:32:573
17349993711,15cyclictest381ktimersoftd/319:20:343
17349993711,15cyclictest381ktimersoftd/319:20:343
1734899376,7cyclictest31-21ksoftirqd/223:30:352
1734899376,7cyclictest31-21ksoftirqd/223:30:352
1734899373,22cyclictest7750irq/56-dwc_otg19:30:322
1734899373,17cyclictest31-21ksoftirqd/221:30:242
17348993716,12cyclictest301ktimersoftd/223:53:282
1734799377,13cyclictest7850irq/56-dwc_otg_19:55:131
1734999367,4cyclictest0-21swapper/322:26:343
1734999366,4cyclictest0-21swapper/300:25:573
1734999363,14cyclictest7850irq/56-dwc_otg_23:05:313
17349993617,7cyclictest7850irq/56-dwc_otg_20:21:473
17349993617,7cyclictest7850irq/56-dwc_otg_20:21:473
17349993615,4cyclictest381ktimersoftd/320:45:223
17349993614,8cyclictest225102chrt20:52:383
17349993614,12cyclictest381ktimersoftd/322:40:143
1734899364,9cyclictest31-21ksoftirqd/222:31:082
1734899363,17cyclictest29-21rcuc/223:42:002
1734899363,14cyclictest29-21rcuc/223:08:022
1734899362,24cyclictest301ktimersoftd/222:20:242
17348993618,8cyclictest301ktimersoftd/222:55:252
1734799362,12cyclictest20862-21ls19:20:121
17349993522,4cyclictest7750irq/56-dwc_otg23:48:033
1734999351,9cyclictest0-21swapper/300:24:313
1734999351,9cyclictest0-21swapper/300:24:313
17349993513,6cyclictest381ktimersoftd/300:10:363
17349993513,4cyclictest27272-21latency_hist23:45:123
17349993511,11cyclictest7950irq/56-dwc_otg_23:39:323
1734899356,15cyclictest301ktimersoftd/200:15:562
1734899355,16cyclictest10499-21ssh22:05:382
1734899355,10cyclictest29-21rcuc/219:41:472
17348993516,10cyclictest7750irq/56-dwc_otg23:29:372
17348993515,5cyclictest17615-21aten_rbpower_en20:40:252
1734899351,24cyclictest0-21swapper/222:40:252
1734799357,11cyclictest19003-21chrt20:43:511
1734799356,15cyclictest0-21swapper/120:11:011
1734999345,14cyclictest7950irq/56-dwc_otg_21:30:273
1734999343,13cyclictest1130-21apt-key19:55:123
17349993416,5cyclictest381ktimersoftd/321:35:323
17349993415,7cyclictest7950irq/56-dwc_otg_21:05:153
1734899349,17cyclictest0-21swapper/221:50:282
1734899347,18cyclictest29-21rcuc/223:59:292
1734899345,6cyclictest29-21rcuc/222:19:592
1734899343,9cyclictest29-21rcuc/219:20:122
1734899342,5cyclictest7750irq/56-dwc_otg20:50:282
1734899342,14cyclictest0-21swapper/222:54:112
1734799347,12cyclictest0-21swapper/120:51:371
1734799341,21cyclictest0-21swapper/119:36:281
1734999331,18cyclictest0-21swapper/322:56:053
1734899338,11cyclictest149-21systemd-journal00:29:092
17348993316,9cyclictest301ktimersoftd/223:47:432
17347993320,10cyclictest22977-21apt-get20:55:241
1734999329,4cyclictest20809-21latency_hist19:20:123
1734999329,12cyclictest26112-21taskset21:02:283
1734999324,15cyclictest4471-21chrt20:02:553
1734999322,13cyclictest7850irq/56-dwc_otg_20:35:223
1734999321,5cyclictest0-21swapper/322:19:373
17349993213,8cyclictest381ktimersoftd/323:16:483
1734899322,7cyclictest0-21swapper/219:45:392
17348993226,4cyclictest10418-21aten_rbpower_po20:20:252
17348993226,4cyclictest10418-21aten_rbpower_po20:20:252
1734899321,21cyclictest0-21swapper/219:50:242
1734999318,6cyclictest381ktimersoftd/319:41:253
1734999317,5cyclictest7750irq/56-dwc_otg20:10:263
1734899317,15cyclictest29-21rcuc/219:39:012
1734899311,7cyclictest12482-21runrttasks20:25:282
1734899311,7cyclictest12482-21runrttasks20:25:282
1734799313,13cyclictest26405-21apt-config21:05:121
1734799312,10cyclictest0-21swapper/119:33:301
17347993111,5cyclictest8501-21/usr/sbin/munin20:15:371
1734999303,4cyclictest0-21swapper/320:57:223
1734999303,12cyclictest7950irq/56-dwc_otg_19:38:503
1734999302,8cyclictest7950irq/56-dwc_otg_19:10:153
17349993011,8cyclictest7950irq/56-dwc_otg_19:49:483
17349993010,8cyclictest180722chrt20:40:313
1734799306,4cyclictest7950irq/56-dwc_otg_19:40:411
1734999294,4cyclictest7950irq/56-dwc_otg_20:30:273
1734999291,5cyclictest7850irq/56-dwc_otg_20:05:243
1734999283,3cyclictest0-21swapper/320:16:243
17349992811,9cyclictest12812-21ntp_kernel_pll_20:25:333
17349992811,9cyclictest12812-21ntp_kernel_pll_20:25:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional