You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 15:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Fri Apr 03, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2615821700,1sleep30-21swapper/321:16:113
869999966,80cyclictest0-21swapper/000:00:270
8699999533,23cyclictest0-21swapper/023:35:270
8699998525,33cyclictest22630-21latency_hist23:10:150
122712820,3sleep2301ktimersoftd/200:15:342
869999811,55cyclictest0-21swapper/022:30:260
869999806,23cyclictest0-21swapper/022:35:190
232142780,4sleep2870799cyclictest21:13:282
165852780,2sleep00-21swapper/021:44:500
8699997732,22cyclictest0-21swapper/019:10:150
276452760,2sleep10-21swapper/120:00:331
869999756,52cyclictest25097-21munin-plugin-st21:55:140
201862750,3sleep320187-21date21:10:153
869999740,54cyclictest7105-21/usr/sbin/munin23:30:270
869999731,39cyclictest0-21swapper/000:30:280
8708997127,18cyclictest25267-21cut21:55:143
869999714,57cyclictest2422-21fschecks_time22:45:320
869999714,57cyclictest2422-21fschecks_time22:45:320
869999713,58cyclictest4502-21latency_hist21:30:140
869999701,48cyclictest27937-21fschecks_time23:55:300
8699997013,4cyclictest3484-21aten_rbpower_en00:05:280
8699996933,19cyclictest1213-21head20:15:410
90402670,2sleep00-21swapper/022:53:270
870799666,14cyclictest7950irq/56-dwc_otg_23:50:132
869999669,46cyclictest9-21ksoftirqd/022:20:280
869999663,53cyclictest18305-21fschecks_time19:35:300
8699996614,5cyclictest101ktimersoftd/020:10:410
8699996610,32cyclictest0-21swapper/021:05:260
8699996610,32cyclictest0-21swapper/021:05:260
178742660,2sleep2301ktimersoftd/222:25:272
95682640,2sleep20-21swapper/223:33:102
8708996424,22cyclictest16754-21ssh23:02:253
869999642,41cyclictest18363-21diskmemload23:05:140
8699996412,48cyclictest101ktimersoftd/020:35:130
8699996410,41cyclictest16154-21fschecks_time00:20:300
26512640,1sleep10-21swapper/121:27:041
8708996328,11cyclictest15819-21memory23:40:363
8708996328,11cyclictest15819-21memory23:40:363
220322630,3sleep3381ktimersoftd/323:48:173
8708996235,21cyclictest8709-21chrt19:10:163
8708996231,28cyclictest2088950ssh22:28:493
8708996231,19cyclictest32682-21ssh22:42:313
8708996230,12cyclictest5528-21/usr/sbin/munin21:30:383
8708996230,12cyclictest5528-21/usr/sbin/munin21:30:383
8708996225,24cyclictest9233-21ssh00:11:233
8708996225,11cyclictest12612-21taskset20:49:213
869999628,32cyclictest22949-21chrt00:28:480
869999623,50cyclictest5362-21apt-get20:30:130
869999621,27cyclictest0-21swapper/023:40:270
869999621,27cyclictest0-21swapper/023:40:270
8708996128,11cyclictest19254-21taskset23:05:333
8708996127,18cyclictest6765-21sh00:09:273
8703996139,16cyclictest8705-21modprobe19:10:151
869999618,38cyclictest0-21swapper/021:55:390
869999615,44cyclictest32173-21rm21:24:190
8699996130,5cyclictest7750irq/56-dwc_otg21:40:140
869999611,15cyclictest0-21swapper/019:55:270
8708996030,13cyclictest12507-21ssh23:35:583
8708996023,10cyclictest26249-21ssh00:32:073
8703996028,15cyclictest23433-21apt-get23:50:251
869999608,40cyclictest21777-21ssh21:10:570
8699996025,9cyclictest7750irq/56-dwc_otg19:10:150
8699996017,26cyclictest12274-21unixbench_singl20:45:400
8699996010,34cyclictest4688-21ssh21:30:230
8699996010,34cyclictest4688-21ssh21:30:230
221632600,3sleep121276-21/usr/sbin/munin21:50:391
108622600,2sleep00-21swapper/000:14:230
8708995933,16cyclictest6009-21ssh23:29:143
8708995931,14cyclictest9590-21ssh23:33:133
8708995925,10cyclictest16316-21ssh00:20:323
869999596,33cyclictest0-21swapper/023:10:530
8708995833,17cyclictest14028-21ssh22:59:593
8708995830,14cyclictest29842-21ssh22:39:543
8708995827,9cyclictest0-21swapper/322:34:213
8708995826,17cyclictest10459-21rm21:36:113
8708995823,12cyclictest28791-21taskset20:05:193
8703995818,19cyclictest16399-21fschecks_count19:30:291
869999589,39cyclictest12662-21kworker/u8:222:25:400
869999582,28cyclictest17291-21unixbench_singl19:30:400
8699995814,41cyclictest101ktimersoftd/023:22:180
8708995731,9cyclictest8206-21ntp_states20:35:363
8708995728,15cyclictest22008-21ssh21:11:213
8708995727,17cyclictest28034-21ssh21:58:043
8708995723,18cyclictest28422-21ps23:55:373
8703995721,17cyclictest26593-21ssh21:16:551
8703995719,12cyclictest650-21ssh23:22:021
8703995716,14cyclictest799250ssh22:51:321
869999578,37cyclictest9409-21cat22:15:140
869999577,35cyclictest5135-21sh23:27:380
869999576,41cyclictest0-21swapper/023:19:100
869999574,16cyclictest28284-21ssh00:35:240
8699995737,4cyclictest7750irq/56-dwc_otg21:15:380
869999572,28cyclictest0-21swapper/019:50:270
869999572,28cyclictest0-21swapper/019:50:270
8708995633,11cyclictest26711-21sshd19:59:533
8708995632,12cyclictest31171-21ssh21:22:273
8708995628,12cyclictest695-21aten_rbpower_vo21:25:283
8708995628,11cyclictest32490-21rm23:20:553
8708995625,12cyclictest14924-21unixbench-2d22:20:443
8708995624,8cyclictest24368-21ssh23:11:123
8708995623,13cyclictest14106-21sh00:17:453
8708995622,13cyclictest23384-21ssh23:50:203
8707995612,16cyclictest18661-21ssh23:05:272
8703995621,13cyclictest29786-21ls20:05:401
8703995616,12cyclictest30422-21taskset00:37:121
8703995612,22cyclictest7950irq/56-dwc_otg_22:20:311
869999565,35cyclictest15354-21ssh00:20:020
8699995633,4cyclictest7750irq/56-dwc_otg21:50:130
869999562,31cyclictest23817-21fschecks_time23:50:300
869999562,30cyclictest0-21swapper/019:45:140
8699995610,25cyclictest0-21swapper/022:40:340
8708995528,15cyclictest2884-21ssh22:06:153
8708995528,12cyclictest4847-21ssh22:48:223
8708995528,12cyclictest4847-21ssh22:48:223
8708995527,15cyclictest31926-21ssh00:39:573
8703995518,15cyclictest21258-21ssh23:08:041
8703995514,12cyclictest7343-21ssh21:33:041
8703995514,12cyclictest7343-21ssh21:33:041
869999557,34cyclictest0-21swapper/022:00:290
869999557,34cyclictest0-21swapper/022:00:290
8699995530,10cyclictest0-21swapper/023:45:320
8708995429,11cyclictest18391-21apt-get21:05:273
8708995422,9cyclictest7950irq/56-dwc_otg_19:49:093
8708995418,23cyclictest7750irq/56-dwc_otg23:19:393
8703995444,6cyclictest221ktimersoftd/122:15:261
8703995417,20cyclictest15900-21apt-config19:30:141
8703995416,22cyclictest9897-21ssh00:12:371
8703995414,20cyclictest15777-21/usr/sbin/munin00:20:321
869999547,26cyclictest0-21swapper/022:55:390
869999544,42cyclictest0-21swapper/022:17:390
869999544,39cyclictest0-21swapper/022:09:170
869999542,27cyclictest0-21swapper/021:00:300
8708995331,12cyclictest19969-21diskmemload22:51:333
8708995329,8cyclictest16620-21irqstats19:30:323
8708995328,10cyclictest9011-21ssh22:14:503
8708995325,16cyclictest20645-21ssh21:49:413
8708995323,11cyclictest15589-21seq19:27:283
8708995320,9cyclictest7950irq/56-dwc_otg_21:00:283
8708995319,18cyclictest7950irq/56-dwc_otg_22:00:593
8708995319,18cyclictest7950irq/56-dwc_otg_22:00:593
8703995319,16cyclictest28809-21ssh23:16:461
869999538,32cyclictest0-21swapper/020:54:260
8699995332,6cyclictest12513-21aten_rbpower_en19:20:260
8708995241,8cyclictest12935-21apt-get20:50:233
8708995226,11cyclictest15748-21ssh21:43:173
8708995223,11cyclictest24900-21runrttasks19:54:493
8708995223,11cyclictest24900-21runrttasks19:54:493
8708995222,13cyclictest18642-21/usr/sbin/munin19:35:353
870399529,26cyclictest599-21ssh22:43:121
8703995217,21cyclictest28094-21/usr/sbin/munin23:55:321
8703995215,28cyclictest14693-21ssh21:41:231
8703995215,24cyclictest19151-21sh21:46:571
8703995214,11cyclictest28162-21ssh22:36:501
869999527,34cyclictest0-21swapper/020:21:430
869999527,34cyclictest0-21swapper/020:21:430
8708995141,7cyclictest12457-21apt-get19:20:233
8708995121,12cyclictest19986-21df_inode19:40:293
8708995119,7cyclictest32169-21apt-get20:15:163
870399514,31cyclictest4671-21ssh22:09:301
8703995122,9cyclictest0-21swapper/123:33:361
8703995120,11cyclictest23879-21rm22:31:441
8703995116,11cyclictest25339-21ssh21:55:171
8703995115,15cyclictest19558-21sh22:26:231
8703995115,10cyclictest7950irq/56-dwc_otg_19:35:381
869999515,36cyclictest7494-21/usr/sbin/munin20:35:400
226792510,2sleep30-21swapper/300:28:193
870399509,24cyclictest0-21swapper/120:13:551
8703995021,14cyclictest21448-21ssh00:26:031
869999509,38cyclictest101ktimersoftd/020:05:390
869999501,25cyclictest0-21swapper/020:00:230
8708994935,11cyclictest11364-21ssh22:16:333
8708994935,11cyclictest10868-21/usr/sbin/munin19:15:303
8708994935,11cyclictest10868-21/usr/sbin/munin19:15:303
870399497,10cyclictest7950irq/56-dwc_otg_23:00:301
8703994919,19cyclictest25884-21rm23:13:581
8703994919,16cyclictest11588-21ssh00:15:191
8703994918,16cyclictest14020-21ssh22:59:581
8703994915,7cyclictest16593-21apt-get21:00:231
8703994911,20cyclictest7850irq/56-dwc_otg_21:14:131
8708994834,11cyclictest30807-21/usr/sbin/munin20:10:283
8708994833,12cyclictest2445-21ls20:20:323
8708994833,12cyclictest2445-21ls20:20:323
870799485,4cyclictest0-21swapper/220:05:262
870799484,7cyclictest7950irq/56-dwc_otg_20:52:312
870399488,27cyclictest0-21swapper/120:43:351
8703994819,17cyclictest4571-21ssh23:26:361
8703994819,12cyclictest31129-21ssh21:22:231
8703994817,11cyclictest1-21systemd22:00:531
8703994817,11cyclictest1-21systemd22:00:531
8703994815,19cyclictest14097-21ssh23:38:521
8703994810,24cyclictest0-21swapper/120:30:201
869999484,36cyclictest0-21swapper/020:57:130
8699994828,4cyclictest0-21swapper/019:45:310
8699994822,12cyclictest0-21swapper/019:15:240
8699994822,12cyclictest0-21swapper/019:15:240
8699994815,5cyclictest101ktimersoftd/020:40:410
8708994733,12cyclictest27799-21/usr/sbin/munin20:00:373
8708994722,11cyclictest812-21ssh00:01:073
8708994719,14cyclictest7950irq/56-dwc_otg_20:30:153
8703994719,16cyclictest23037-21taskset19:49:031
8703994715,12cyclictest4774-21ssh22:48:141
8703994715,12cyclictest4774-21ssh22:48:141
8707994622,5cyclictest16317-21/usr/sbin/munin00:20:322
870399467,25cyclictest0-21swapper/120:35:261
8703994618,17cyclictest18520-21ssh23:44:251
8703994618,17cyclictest18520-21ssh23:44:251
8703994616,15cyclictest23000-21ssh23:50:021
8703994612,10cyclictest812-21ssh00:01:071
8708994530,12cyclictest9887-21memory20:40:343
8708994530,12cyclictest4804-21/usr/sbin/munin20:25:413
870399456,19cyclictest0-21swapper/119:40:251
8703994518,16cyclictest10480-21ssh21:36:141
8703994517,13cyclictest19586-21chrt21:05:501
8703994517,13cyclictest19586-21chrt21:05:501
8703994516,19cyclictest7506-21ssh22:12:061
869999456,28cyclictest0-21swapper/019:27:240
8708994429,11cyclictest15216-21if_wlan020:55:313
870799443,9cyclictest2781-21ssh23:25:272
870399447,23cyclictest0-21swapper/120:25:201
870399447,18cyclictest0-21swapper/120:55:261
870799432,8cyclictest0-21swapper/220:10:262
8703994313,12cyclictest5238-21ssh00:06:391
8703994310,10cyclictest7850irq/56-dwc_otg_00:32:031
8703994127,12cyclictest11044-21if_eth019:15:321
8703994127,12cyclictest11044-21if_eth019:15:321
870799407,22cyclictest0-21swapper/219:55:272
870799403,16cyclictest7850irq/56-dwc_otg_19:10:152
8707994020,18cyclictest0-21swapper/220:30:262
870799401,32cyclictest0-21swapper/220:25:282
870399407,23cyclictest0-21swapper/120:52:321
8703994014,11cyclictest3189-21taskset20:22:041
8703994014,11cyclictest3189-21taskset20:22:041
84862406,21sleep18606-21grep19:10:151
8703993912,10cyclictest0-21swapper/119:54:321
8703993912,10cyclictest0-21swapper/119:54:321
322642390,2sleep20-21swapper/222:41:452
128542390,4sleep231-21ksoftirqd/222:57:522
870799388,14cyclictest7950irq/56-dwc_otg_21:45:262
870799381,20cyclictest0-21swapper/200:13:152
870399389,11cyclictest0-21swapper/120:19:211
72432385,21sleep37330-21/usr/sbin/munin19:05:313
870799378,12cyclictest7850irq/56-dwc_otg_00:08:312
8707993718,4cyclictest908-21runrttasks00:37:232
8707993716,4cyclictest2468450wget21:15:292
870799371,23cyclictest23706-21cut19:50:312
870799371,23cyclictest23706-21cut19:50:312
8703993715,11cyclictest149-21systemd-journal19:24:441
258962370,2sleep231-21ksoftirqd/221:55:312
870799362,4cyclictest7850irq/56-dwc_otg_19:30:242
870399368,9cyclictest7850irq/56-dwc_otg_20:46:411
164122361,30sleep216497-21ssh23:40:442
164122361,30sleep216497-21ssh23:40:442
870799358,6cyclictest31-21ksoftirqd/223:20:402
870799358,13cyclictest7950irq/56-dwc_otg_22:00:302
870799358,13cyclictest7950irq/56-dwc_otg_22:00:302
870799358,12cyclictest7950irq/56-dwc_otg_22:05:322
870799353,17cyclictest7950irq/56-dwc_otg_21:40:292
870799352,8cyclictest7850irq/56-dwc_otg_19:20:402
870799351,24cyclictest0-21swapper/200:00:302
870799351,23cyclictest23738-21cut23:50:302
8707993510,16cyclictest7750irq/56-dwc_otg22:20:282
870799345,21cyclictest0-21swapper/223:04:372
8707993423,7cyclictest301ktimersoftd/223:15:262
870799341,8cyclictest19536-21grep21:05:432
870799341,8cyclictest19536-21grep21:05:432
8707993415,7cyclictest149-21systemd-journal19:35:302
870799341,24cyclictest24564-21fschecks_time00:30:342
870799341,11cyclictest0-21swapper/223:58:182
8703993419,11cyclictest26300-21sed19:55:411
870799339,3cyclictest7750irq/56-dwc_otg22:50:212
870799334,5cyclictest30142-21latency_hist22:40:142
870799332,19cyclictest0-21swapper/221:25:012
870799332,15cyclictest0-21swapper/220:35:292
870799331,5cyclictest7950irq/56-dwc_otg_22:33:282
870799331,22cyclictest2749-21missed_timers22:45:362
870799331,22cyclictest2749-21missed_timers22:45:362
870799331,13cyclictest7750irq/56-dwc_otg20:40:282
870799328,11cyclictest7850irq/56-dwc_otg_21:35:292
870799321,27cyclictest4948-21ls21:30:292
870799321,27cyclictest4948-21ls21:30:292
870799321,24cyclictest9950-21uniq22:15:322
870799319,10cyclictest7950irq/56-dwc_otg_21:50:272
870799318,12cyclictest0-21swapper/223:15:142
870799317,7cyclictest0-21swapper/200:25:252
870799314,15cyclictest7950irq/56-dwc_otg_23:39:012
870799313,6cyclictest7750irq/56-dwc_otg19:45:302
870799311,4cyclictest0-21swapper/220:55:302
870799303,5cyclictest7750irq/56-dwc_otg22:10:332
870799302,6cyclictest7850irq/56-dwc_otg_20:00:392
8707993022,5cyclictest1009-21sh21:25:332
870799302,14cyclictest7950irq/56-dwc_otg_19:17:252
870799302,14cyclictest7950irq/56-dwc_otg_19:17:252
870799292,6cyclictest7750irq/56-dwc_otg19:25:302
8707992920,6cyclictest213212sleep219:45:142
8707992912,10cyclictest7750irq/56-dwc_otg21:00:332
870799271,12cyclictest7950irq/56-dwc_otg_20:24:032
870799271,12cyclictest7950irq/56-dwc_otg_20:24:032
870799262,6cyclictest31-21ksoftirqd/220:15:152
870799261,15cyclictest0-21swapper/220:45:212
68672263,14sleep20-21swapper/219:05:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional