You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 05:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Jul 20, 2026 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2196991201,95cyclictest0-21swapper/022:35:420
2196991201,95cyclictest0-21swapper/022:35:420
2941421050,6sleep1219799cyclictest22:08:401
2196991056,54cyclictest0-21swapper/021:30:350
2196991056,54cyclictest0-21swapper/021:30:350
21969910447,15cyclictest22320-21ssh00:35:500
21969910117,55cyclictest0-21swapper/023:15:450
219699999,58cyclictest20753-21fschecks_count23:55:380
2196999626,53cyclictest7167-21grep20:50:490
2196999624,60cyclictest22083-21fschecks_time21:20:440
219799953,9cyclictest8050irq/56-dwc_otg_22:35:421
219799953,9cyclictest8050irq/56-dwc_otg_22:35:421
2196999527,51cyclictest8804-21diskmemload23:01:090
2196999439,38cyclictest8710-21ntpq23:40:440
2196999332,41cyclictest3936-21aten_rbpower_vo23:35:350
2196999322,56cyclictest7904-21fschecks_count19:25:380
219899924,7cyclictest8050irq/56-dwc_otg_21:55:432
219699923,67cyclictest0-21swapper/021:55:440
2196999129,44cyclictest31248-21latency22:10:400
2196999129,44cyclictest31248-21latency22:10:400
2196999127,14cyclictest27000-21grep20:15:490
2196999127,14cyclictest27000-21grep20:15:490
2196998926,51cyclictest18636-21awk22:35:210
2196998920,54cyclictest0-21swapper/022:55:290
219799885,23cyclictest0-21swapper/123:15:441
2196998828,45cyclictest10655-21unixbench_singl19:30:510
219699882,59cyclictest0-21swapper/022:05:410
219799873,16cyclictest7950irq/56-dwc_otg_22:00:411
219699873,14cyclictest0-21swapper/022:00:390
2196998727,50cyclictest20228-21sh21:18:210
2196998726,41cyclictest17089-21aten_rbpower_po00:30:350
219899864,75cyclictest20769-21fschecks_time23:55:382
2196998633,48cyclictest5503-21ssh21:40:200
2196998627,45cyclictest10930-21ssh21:45:450
2196998625,44cyclictest3430-21timerandwakeup22:15:490
2197998510,8cyclictest8050irq/56-dwc_otg_21:10:391
219699855,15cyclictest8050irq/56-dwc_otg_22:52:210
219899843,75cyclictest22180-21timerandwakeup00:35:472
219899843,10cyclictest8726-21ntp_states23:40:462
2196998425,48cyclictest0-21swapper/000:10:370
2196998422,55cyclictest0-21swapper/000:00:480
2196998335,41cyclictest8050irq/56-dwc_otg_23:10:400
2196998328,33cyclictest13037-21diskmemload22:26:520
2196998323,48cyclictest0-21swapper/022:24:050
2196998321,51cyclictest0-21swapper/023:09:430
2196998318,58cyclictest0-21swapper/023:32:430
219899823,11cyclictest19726-21ssh22:35:452
219899823,11cyclictest19726-21ssh22:35:452
219799825,14cyclictest0-21swapper/100:35:491
219699824,25cyclictest13633-21ssh21:10:380
2196998233,34cyclictest29077-21ssh22:46:480
2196998227,12cyclictest5706-21cat19:20:200
2196998222,45cyclictest2201-21seq19:10:220
219799814,16cyclictest18956-21memory21:55:431
2196998123,43cyclictest0-21swapper/020:45:460
219899803,7cyclictest0-21swapper/222:05:422
219799803,12cyclictest20830-21fschecks_time23:55:391
2196998034,10cyclictest24054-21apt-get20:10:320
2196998030,30cyclictest17058-21taskset21:53:410
2196998023,43cyclictest0-21swapper/021:43:030
219999793,55cyclictest0-21swapper/323:10:403
219899792,13cyclictest16079-21if_wlan023:10:402
219799795,18cyclictest16088-21ssh23:10:401
219799795,11cyclictest0-21swapper/123:40:431
2196997923,29cyclictest0-21swapper/023:23:560
2196997923,29cyclictest0-21swapper/023:23:560
219899783,69cyclictest23030-21taskset22:00:402
219899782,9cyclictest13657-21ssh21:10:402
2196997824,40cyclictest0-21swapper/021:26:130
2196997823,37cyclictest5836-21dhclient19:20:240
2196997821,40cyclictest7408-21rm00:18:360
2196997741,20cyclictest22255-21apt-get20:05:370
2196997728,10cyclictest2217-21cron20:40:200
2196997725,35cyclictest0-21swapper/000:30:150
2196997725,11cyclictest16203-21apt-get23:50:330
2196997723,35cyclictest0-21swapper/000:07:230
2196997723,32cyclictest12766-21runrttasks21:07:050
2196997723,32cyclictest12766-21runrttasks21:07:050
219899763,14cyclictest20479-21ntp_states23:15:442
2196997621,47cyclictest13037-21diskmemload00:21:430
2196997529,11cyclictest24573-21ssh22:41:080
2196997525,36cyclictest0-21swapper/020:43:030
2196997521,44cyclictest0-21swapper/020:26:560
2196997519,40cyclictest0-21swapper/023:27:130
2196997518,48cyclictest0-21swapper/021:01:370
2196997323,32cyclictest9308-21taskset20:58:560
2196997322,39cyclictest0-21swapper/023:50:080
2196997024,36cyclictest0-21swapper/020:31:560
2196997022,36cyclictest0-21swapper/019:57:130
2196997022,36cyclictest0-21swapper/019:57:130
2196997022,32cyclictest14571-21taskset19:43:590
2199996817,18cyclictest381ktimersoftd/323:15:453
219899683,13cyclictest1086-21ssh22:52:232
219799684,6cyclictest7850irq/56-dwc_otg23:50:211
2196996824,29cyclictest0-21swapper/019:36:320
2196996820,33cyclictest0-21swapper/020:23:530
2196996619,35cyclictest0-21swapper/019:51:540
52682650,2sleep10-21swapper/100:15:431
2199996521,18cyclictest7950irq/56-dwc_otg_22:35:443
2199996521,18cyclictest7950irq/56-dwc_otg_22:35:443
2199996511,33cyclictest8050irq/56-dwc_otg_21:55:453
219899652,10cyclictest6049-21ssh22:58:522
219799651,6cyclictest7850irq/56-dwc_otg20:45:391
219899644,49cyclictest1698-21sed00:10:502
2198996319,41cyclictest1341-21latency_hist21:35:212
2198996319,41cyclictest1341-21latency_hist21:35:212
219899631,14cyclictest8050irq/56-dwc_otg_20:05:452
2196996322,23cyclictest21882-21taskset20:05:150
3692620,1sleep10-21swapper/121:33:421
3692620,1sleep10-21swapper/121:33:421
219999626,32cyclictest0-21swapper/322:52:223
2199996233,23cyclictest0-21swapper/300:35:493
2199996214,29cyclictest0-21swapper/321:10:383
219899628,45cyclictest24544-21taskset21:23:402
2196996228,31cyclictest162142sleep019:47:010
219899611,7cyclictest4522-21sort20:45:352
219899611,6cyclictest6872-21apt-get23:00:272
219899611,10cyclictest31770-21awk23:30:202
2197996110,4cyclictest0-21swapper/121:52:531
254202600,5sleep3219999cyclictest22:42:403
2199996018,22cyclictest8705-21awk23:40:443
219799604,5cyclictest0-21swapper/120:10:431
219999596,22cyclictest0-21swapper/321:30:353
219999596,22cyclictest0-21swapper/321:30:353
2199995913,33cyclictest0-21swapper/322:05:403
2199995910,11cyclictest7950irq/56-dwc_otg_20:10:333
219899595,45cyclictest0-21swapper/200:00:512
219899594,5cyclictest8811-21grep19:25:492
219899593,12cyclictest0-21swapper/220:40:342
2198995911,40cyclictest0-21swapper/221:53:532
219899591,13cyclictest0-21swapper/222:10:492
219899591,13cyclictest0-21swapper/222:10:492
219899591,13cyclictest0-21swapper/221:15:452
219899591,12cyclictest18295-21sed00:30:502
219999583,39cyclictest0-21swapper/322:00:413
219999581,43cyclictest8050irq/56-dwc_otg_21:54:143
219899582,10cyclictest5902-21aten_rbpower_en19:20:352
219899581,11cyclictest0-21swapper/219:15:412
219799581,17cyclictest7950irq/56-dwc_otg_21:20:551
2197995810,4cyclictest0-21swapper/122:11:401
2197995810,4cyclictest0-21swapper/122:11:401
38172570,2sleep20-21swapper/223:35:342
219899571,4cyclictest0-21swapper/223:50:382
219899571,12cyclictest28696-21taskset21:28:412
219899563,3cyclictest0-21swapper/200:21:322
219899562,13cyclictest0-21swapper/221:40:072
219899561,6cyclictest24550-21latency20:10:412
2198995612,40cyclictest0-21swapper/221:05:352
2198995612,40cyclictest0-21swapper/221:05:352
2199995522,10cyclictest20860-21cat23:55:403
219899559,35cyclictest0-21swapper/222:24:382
219899556,4cyclictest8050irq/56-dwc_otg_20:15:432
219899556,4cyclictest8050irq/56-dwc_otg_20:15:432
219899546,7cyclictest0-21swapper/220:30:372
219899545,10cyclictest125-21systemd-journal19:46:502
219899543,8cyclictest0-21swapper/200:27:562
219899541,9cyclictest0-21swapper/219:10:322
219899541,8cyclictest0-21swapper/222:44:302
219899541,5cyclictest5830-21taskset21:40:262
219899541,12cyclictest11407-21ssh21:45:562
219899537,35cyclictest0-21swapper/200:15:452
219899534,9cyclictest13440-21ssh23:07:032
219899533,4cyclictest8050irq/56-dwc_otg_22:15:342
219799533,26cyclictest0-21swapper/122:52:221
2197995315,7cyclictest8050irq/56-dwc_otg_23:10:161
219899528,3cyclictest8050irq/56-dwc_otg_00:05:532
219899521,8cyclictest0-21swapper/220:37:302
2198995211,4cyclictest32485-21tune2fs23:30:392
219899521,14cyclictest0-21swapper/222:33:032
219799529,31cyclictest0-21swapper/100:30:341
219899513,8cyclictest182402sleep219:54:442
219899512,6cyclictest0-21swapper/222:47:232
219899511,5cyclictest27654-21awk23:25:212
219899511,5cyclictest27654-21awk23:25:212
219899511,11cyclictest0-21swapper/220:59:272
219899508,6cyclictest8050irq/56-dwc_otg_21:04:522
219899506,4cyclictest7950irq/56-dwc_otg_19:36:162
219899502,4cyclictest0-21swapper/220:05:202
219899501,3cyclictest8050irq/56-dwc_otg_23:50:212
219899493,5cyclictest19857-21taskset19:57:242
219899493,5cyclictest19857-21taskset19:57:242
219899491,10cyclictest12659-21diskmemload22:27:012
2197994919,12cyclictest7950irq/56-dwc_otg_21:45:311
219999483,20cyclictest8050irq/56-dwc_otg_22:15:493
219999482,7cyclictest7950irq/56-dwc_otg_23:05:333
219999472,3cyclictest7950irq/56-dwc_otg_23:20:353
219999472,3cyclictest7950irq/56-dwc_otg_23:20:353
219899479,5cyclictest10657-21grep19:30:512
219899471,5cyclictest7300-21chrt20:51:192
219799471,20cyclictest7718-21ls23:00:401
219999468,23cyclictest0-21swapper/320:40:333
219999467,25cyclictest0-21swapper/319:30:363
219899464,5cyclictest30728-21runrttasks20:26:442
219899461,10cyclictest28932-21taskset20:21:442
219799466,22cyclictest8050irq/56-dwc_otg_23:00:211
219999456,25cyclictest27992-21ssh23:25:353
219999441,5cyclictest381ktimersoftd/321:40:203
219899441,10cyclictest0-21swapper/219:44:102
17042443,12sleep00-21swapper/019:07:320
17042443,12sleep00-21swapper/019:07:320
219799431,5cyclictest1651-21ssh00:10:501
2197994314,4cyclictest0-21swapper/123:30:421
219999427,23cyclictest0-21swapper/320:05:253
219999427,10cyclictest39-21ksoftirqd/322:10:403
219999427,10cyclictest39-21ksoftirqd/322:10:403
148192420,2sleep30-21swapper/322:30:323
3652410,2sleep30-21swapper/323:30:443
219999418,20cyclictest37-21rcuc/300:13:343
2199994113,5cyclictest381ktimersoftd/319:25:433
2199994110,10cyclictest7850irq/56-dwc_otg21:45:473
219999404,13cyclictest8050irq/56-dwc_otg_19:20:493
2199994018,5cyclictest37-21rcuc/322:21:173
2199994015,5cyclictest9550irq/48-DMA15:26:473
219799401,15cyclictest14164-21ssh00:25:491
219999398,16cyclictest39-21ksoftirqd/319:15:363
219999396,16cyclictest37-21rcuc/323:36:033
219999392,18cyclictest37-21rcuc/322:55:313
219799394,7cyclictest0-21swapper/122:29:361
219799392,4cyclictest8050irq/56-dwc_otg_20:30:331
219799392,26cyclictest3590-21grep22:15:501
219999387,21cyclictest0-21swapper/321:40:263
219999382,9cyclictest0-21swapper/300:05:383
219999382,18cyclictest37-21rcuc/320:21:353
2199993814,8cyclictest8050irq/56-dwc_otg_00:30:383
219999381,20cyclictest39-21ksoftirqd/323:53:043
2199993810,10cyclictest7850irq/56-dwc_otg22:25:373
219799381,7cyclictest3194850ssh00:09:191
2197993812,7cyclictest21352-21idleruntime-cro21:20:201
219999379,6cyclictest7850irq/56-dwc_otg20:30:503
219999373,16cyclictest8050irq/56-dwc_otg_22:50:203
2199993727,7cyclictest21933-21latency_hist20:05:203
2199993716,5cyclictest12031-21sh00:24:283
219999371,15cyclictest37-21rcuc/321:20:463
219799375,4cyclictest0-21swapper/122:30:331
219799372,13cyclictest24912-21fschecks_time00:00:411
219799371,27cyclictest9776-21cut19:30:391
219999369,9cyclictest7850irq/56-dwc_otg00:30:163
2199993611,14cyclictest0-21swapper/323:45:423
219799368,13cyclictest0-21swapper/123:50:481
219999357,7cyclictest37-21rcuc/321:15:393
219999355,17cyclictest37-21rcuc/323:04:063
219999354,6cyclictest4262-21apt-get20:45:303
219999351,7cyclictest0-21swapper/319:35:353
219999351,23cyclictest25645-21timerandwakeup00:00:503
219799352,5cyclictest7850irq/56-dwc_otg20:40:481
169562350,2sleep10-21swapper/119:50:361
219999346,15cyclictest0-21swapper/320:50:343
219999341,22cyclictest39-21ksoftirqd/319:52:053
219999341,22cyclictest15212-21cut19:45:363
2199993410,10cyclictest7850irq/56-dwc_otg00:16:023
219799349,10cyclictest0-21swapper/122:43:151
219799348,11cyclictest0-21swapper/123:37:101
219799347,13cyclictest8050irq/56-dwc_otg_19:35:491
219799346,5cyclictest7850irq/56-dwc_otg23:25:211
219799346,5cyclictest7850irq/56-dwc_otg23:25:211
219799343,14cyclictest8050irq/56-dwc_otg_20:20:221
219799343,14cyclictest8050irq/56-dwc_otg_20:20:221
2197993418,4cyclictest13037-21diskmemload22:45:411
2197993413,9cyclictest28524-21kernelversion23:25:421
219999338,5cyclictest8050irq/56-dwc_otg_19:55:373
219999338,5cyclictest8050irq/56-dwc_otg_19:55:373
219999334,5cyclictest0-21swapper/319:10:343
219999331,16cyclictest37-21rcuc/321:26:163
219799338,5cyclictest0-21swapper/121:38:081
219799337,4cyclictest7950irq/56-dwc_otg_19:10:491
219799337,12cyclictest0-21swapper/121:28:421
219799331,25cyclictest15282-21ls19:45:361
2197993312,4cyclictest29988-21grep20:25:401
103722331,2sleep10-21swapper/100:21:271
4672325,13sleep20-21swapper/219:05:362
4672325,13sleep20-21swapper/219:05:362
219999324,5cyclictest7850irq/56-dwc_otg21:10:213
219999324,5cyclictest7850irq/56-dwc_otg21:10:213
2197993220,9cyclictest0-21swapper/121:40:381
2197993213,5cyclictest727-21runrttasks19:56:201
2197993213,5cyclictest727-21runrttasks19:56:201
219799321,20cyclictest22421-21ls20:05:401
219999319,9cyclictest7950irq/56-dwc_otg_20:40:053
219999315,6cyclictest14276-21grep19:40:503
219999312,21cyclictest0-21swapper/321:04:073
219799316,15cyclictest0-21swapper/120:00:331
219799314,5cyclictest125-21systemd-journal20:55:201
219799313,4cyclictest7950irq/56-dwc_otg_21:00:451
219999304,11cyclictest8050irq/56-dwc_otg_20:55:413
219799309,4cyclictest8050irq/56-dwc_otg_19:40:331
219799306,15cyclictest0-21swapper/121:05:341
219799306,15cyclictest0-21swapper/121:05:341
219799305,6cyclictest8050irq/56-dwc_otg_22:20:491
219799304,7cyclictest7850irq/56-dwc_otg19:25:201
2197993013,4cyclictest611-21/usr/sbin/munin20:35:421
219799295,12cyclictest7850irq/56-dwc_otg20:20:401
219799291,5cyclictest0-21swapper/119:15:461
219799284,4cyclictest7950irq/56-dwc_otg_19:25:391
18722276,14sleep10-21swapper/119:09:441
18722276,14sleep10-21swapper/119:09:441
219999261,5cyclictest26447-21missed_timers20:15:423
219999261,5cyclictest26447-21missed_timers20:15:423
2197992514,5cyclictest7855-21/usr/sbin/munin20:55:391
16722254,14sleep30-21swapper/319:07:063
16722254,14sleep30-21swapper/319:07:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional