You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 19:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Tue Mar 31, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1032621440,5sleep20-21swapper/207:05:292
80012990,2sleep20-21swapper/209:30:272
11869999441,6cyclictest106-21jbd2/mmcblk0p2-11:05:240
1187099906,33cyclictest0-21swapper/109:45:261
1187099906,33cyclictest0-21swapper/109:45:261
1186999876,62cyclictest0-21swapper/010:55:180
1186999856,15cyclictest0-21swapper/010:05:270
1186999837,15cyclictest0-21swapper/010:35:240
11869998165,7cyclictest0-21swapper/009:45:260
11869998165,7cyclictest0-21swapper/009:45:260
154962760,2sleep10-21swapper/110:55:451
1186999761,4cyclictest7750irq/56-dwc_otg09:15:230
1186999761,4cyclictest7750irq/56-dwc_otg09:15:230
222142750,2sleep30-21swapper/312:23:033
11869997550,9cyclictest7750irq/56-dwc_otg12:20:230
191212740,5sleep11187099cyclictest11:00:361
11871997418,45cyclictest24457-21sh11:06:582
1186999731,38cyclictest0-21swapper/008:00:240
299672720,2sleep20-21swapper/211:14:302
299672720,2sleep20-21swapper/211:14:302
11869997231,5cyclictest0-21swapper/007:25:260
109562720,5sleep30-21swapper/307:05:383
1187099717,29cyclictest0-21swapper/107:45:251
11869997121,47cyclictest101ktimersoftd/011:20:260
30112700,6sleep21187199cyclictest08:15:242
1187199708,40cyclictest0-21swapper/210:50:122
1186999703,37cyclictest0-21swapper/011:35:270
1186999701,32cyclictest0-21swapper/010:20:230
1187199694,46cyclictest7950irq/56-dwc_otg_09:00:242
1187199691,50cyclictest7950irq/56-dwc_otg_09:05:372
1187199684,46cyclictest7950irq/56-dwc_otg_07:55:332
1187199682,5cyclictest7478-21/usr/sbin/munin11:25:382
11869996820,26cyclictest9742-21latency_hist10:50:130
1186999681,34cyclictest0-21swapper/007:40:250
11870996727,24cyclictest21234-21/usr/sbin/munin09:05:331
1187199661,51cyclictest0-21swapper/209:43:482
11869996610,12cyclictest0-21swapper/009:25:260
75552650,2sleep10-21swapper/108:25:411
1187199651,48cyclictest0-21swapper/210:59:022
1187199650,3cyclictest7950irq/56-dwc_otg_12:22:522
1187099657,31cyclictest0-21swapper/107:25:261
1186999657,34cyclictest9-21ksoftirqd/012:25:260
11869996553,5cyclictest101ktimersoftd/010:10:350
11869996548,5cyclictest101ktimersoftd/011:45:380
11869996522,7cyclictest7850irq/56-dwc_otg_11:55:230
1187199641,54cyclictest16593-21ssh10:19:082
11870996420,28cyclictest25776-21ssh11:09:241
1187199631,51cyclictest25202-21/usr/sbin/munin07:45:342
11870996319,24cyclictest24840-21ls12:25:391
1186999636,33cyclictest0-21swapper/008:45:270
1186999636,33cyclictest0-21swapper/008:45:270
1187199621,50cyclictest21698-21ssh11:43:162
1187199621,50cyclictest21698-21ssh11:43:162
11870996223,12cyclictest7171-21/usr/sbin/munin12:05:441
11870996221,23cyclictest21451-21date10:25:131
11869996229,20cyclictest0-21swapper/011:00:230
317332610,34sleep21187199cyclictest11:15:392
248002610,2sleep20-21swapper/211:46:232
1187299611,9cyclictest14930-21ntp_states10:55:373
1187199618,39cyclictest0-21swapper/211:22:522
1187199611,50cyclictest27607-21uptime07:50:422
11870996128,12cyclictest8321-21rm11:26:301
11870996124,12cyclictest3199-21ntp_states11:20:351
11870996122,27cyclictest24448-21latency_hist07:45:131
1186999613,4cyclictest0-21swapper/010:25:240
11869996128,12cyclictest7750irq/56-dwc_otg12:35:400
11869996115,24cyclictest15119-21cron12:15:130
11869996115,24cyclictest15119-21cron12:15:130
11869996112,29cyclictest9-21ksoftirqd/007:50:120
1187199601,5cyclictest2687-21ssh09:24:242
1187199601,5cyclictest13355-21apt-get08:41:562
1187199601,49cyclictest11225-21ssh10:11:542
1187199601,44cyclictest7375-21ssh09:30:092
1187199601,40cyclictest0-21swapper/208:05:372
1187199601,40cyclictest0-21swapper/208:05:372
11870996024,19cyclictest19152-21ssh09:43:231
11870996023,22cyclictest29506-21ssh12:31:151
11870996022,16cyclictest2162250ssh11:43:081
11870996022,16cyclictest2162250ssh11:43:081
11870996021,23cyclictest30806-21ssh09:19:181
11870996021,23cyclictest30806-21ssh09:19:181
1186999606,22cyclictest0-21swapper/012:30:240
1186999602,30cyclictest13765-21cut10:15:320
280342590,2sleep20-21swapper/210:32:232
1187199595,38cyclictest939-21lldpd10:43:172
1187199595,38cyclictest939-21lldpd10:43:172
1187199592,41cyclictest908-21runrttasks09:35:322
1187199592,39cyclictest0-21swapper/211:30:532
1187199592,39cyclictest0-21swapper/210:20:552
1187199591,4cyclictest3347-21ssh12:40:112
1187199591,47cyclictest5795-21ssh12:03:152
1187199591,41cyclictest7750irq/56-dwc_otg10:51:452
1187199591,3cyclictest7850irq/56-dwc_otg_10:08:222
11870995923,22cyclictest12875-21ssh09:35:381
11870995923,20cyclictest14762-21ssh12:14:331
11870995923,20cyclictest14762-21ssh12:14:331
11870995921,24cyclictest7428-21munin-run09:30:121
11869995924,10cyclictest0-21swapper/012:00:250
1187199584,37cyclictest7950irq/56-dwc_otg_12:25:412
1187199584,36cyclictest7750irq/56-dwc_otg08:00:392
1187199582,45cyclictest0-21swapper/208:36:462
1187199581,47cyclictest17586-21rm12:17:092
1187199581,44cyclictest884-21ssh11:56:522
1187199581,2cyclictest0-21swapper/212:08:302
11870995827,13cyclictest1213-21ssh11:57:261
11870995825,11cyclictest7622-21ssh10:07:521
11870995823,17cyclictest7812-21ssh10:47:021
11870995822,23cyclictest20172-21/usr/sbin/munin09:00:351
11870995821,9cyclictest11691-21munin-run09:35:121
11870995821,23cyclictest14991-21ssh10:16:131
11870995820,23cyclictest12277-21ssh10:13:501
11869995837,5cyclictest13-21rcuc/008:00:130
1186999581,33cyclictest1867-21awk08:10:340
1186999581,29cyclictest20182-21grep07:30:410
1186999581,28cyclictest26992-21rm10:30:410
1187199571,43cyclictest0-21swapper/212:14:332
1187199571,43cyclictest0-21swapper/212:14:332
1187199571,3cyclictest7850irq/56-dwc_otg_09:16:352
1187199571,3cyclictest7850irq/56-dwc_otg_09:16:352
1187099575,10cyclictest862-21ssh10:00:241
11870995730,12cyclictest20439-21sshd07:32:211
11870995726,12cyclictest26296-21ssh09:13:531
11870995722,19cyclictest24140-21cat11:45:381
11870995722,16cyclictest26241-21if_err_wlan010:30:311
11870995720,27cyclictest587-21runrttasks08:05:421
11870995720,27cyclictest587-21runrttasks08:05:421
11869995730,5cyclictest7750irq/56-dwc_otg09:20:290
1186999572,29cyclictest22945-21diskmemload09:10:130
1186999572,28cyclictest6889-21fschecks_time11:25:300
1187299566,12cyclictest7950irq/56-dwc_otg_10:47:143
1187299565,5cyclictest7950irq/56-dwc_otg_09:30:013
1187199564,36cyclictest7950irq/56-dwc_otg_07:40:272
1187199561,5cyclictest28952-21ssh11:51:252
1187199561,4cyclictest7850irq/56-dwc_otg_09:54:142
1187199561,4cyclictest22074-21ssh09:46:042
1187199561,4cyclictest22074-21ssh09:46:042
1187199561,41cyclictest17825-21df08:55:272
11870995632,8cyclictest17627-21apt-get08:55:231
11870995624,14cyclictest30025-21/usr/sbin/munin08:00:341
11870995624,13cyclictest12780-21ssh11:32:051
1186999564,40cyclictest0-21swapper/009:55:240
1186999564,32cyclictest9-21ksoftirqd/012:05:320
11869995636,5cyclictest101ktimersoftd/009:40:360
11869995629,11cyclictest32640-21processes08:05:370
11869995629,11cyclictest32640-21processes08:05:370
1187199553,4cyclictest7950irq/56-dwc_otg_10:25:242
1187199552,41cyclictest7950irq/56-dwc_otg_11:39:532
1187199552,37cyclictest0-21swapper/212:34:042
1187199551,5cyclictest610-21ssh10:39:012
1187199551,4cyclictest21322-21ssh11:03:502
11871995510,38cyclictest3142-21sshd10:02:212
11870995529,18cyclictest13451-1kworker/u9:112:17:021
11870995525,16cyclictest1724-21ssh12:37:141
11870995523,16cyclictest7950irq/56-dwc_otg_10:35:541
11870995523,14cyclictest2849-21ssh09:24:411
11870995522,21cyclictest28473-21/usr/sbin/munin07:55:291
11870995519,22cyclictest12460-21ssh10:52:571
11870995518,24cyclictest0-21swapper/111:54:351
1186999552,8cyclictest10502-21fschecks_count08:35:290
1186999552,33cyclictest13721-21awk10:55:120
1186999552,32cyclictest5627-21latency_hist10:45:140
1186999552,32cyclictest5627-21latency_hist10:45:140
11869995517,19cyclictest0-21swapper/007:20:330
11869995514,19cyclictest13-21rcuc/011:40:160
11869995514,19cyclictest13-21rcuc/011:40:160
1186999551,33cyclictest27577-21grep07:50:420
1187199544,46cyclictest23158-21ssh09:10:232
1187199541,4cyclictest31037-21ssh09:56:502
11870995427,16cyclictest3028-21sh12:00:201
11870995424,21cyclictest11752-21sshd08:39:391
11870995423,11cyclictest26091-21apt-get07:50:181
11870995419,20cyclictest27343-21ssh09:52:491
11870995418,18cyclictest7750irq/56-dwc_otg09:59:371
11870995417,20cyclictest16995-21sshd07:24:271
11869995415,22cyclictest9-21ksoftirqd/011:15:130
11869995415,22cyclictest9-21ksoftirqd/011:15:130
1187199535,31cyclictest7850irq/56-dwc_otg_07:35:362
1187199531,6cyclictest4826-21/usr/sbin/munin08:20:382
11870995327,12cyclictest17226-21ssh11:37:391
11870995320,16cyclictest24098-21ssh10:27:441
11870995320,14cyclictest5057-21ssh10:44:331
11870995320,14cyclictest5057-21ssh10:44:331
1186999535,30cyclictest0-21swapper/009:35:380
11869995326,12cyclictest7950irq/56-dwc_otg_11:15:290
11869995314,20cyclictest1254-21fschecks_time10:00:310
174302520,1sleep30-21swapper/311:38:023
1187199521,32cyclictest0-21swapper/207:32:212
11869995224,4cyclictest4869-21wget08:20:260
1187199513,38cyclictest7850irq/56-dwc_otg_08:50:122
1187199513,38cyclictest7850irq/56-dwc_otg_08:50:122
1187199511,3cyclictest0-21swapper/208:34:082
11870995123,12cyclictest30235-21ssh11:15:001
11870995123,12cyclictest30235-21ssh11:15:001
11870995122,14cyclictest30752-21tr11:15:261
11870995119,20cyclictest15029-21seq07:17:171
11869995144,5cyclictest101ktimersoftd/009:35:120
1186999511,28cyclictest17711-21grep08:55:260
1187199501,38cyclictest0-21swapper/208:13:402
11870995023,14cyclictest2606-21chrt08:13:131
11870995022,13cyclictest19855-21diskstats12:20:291
11870995021,14cyclictest149-21systemd-journal08:23:571
11870995017,23cyclictest15481-21idleruntime-cro08:50:121
11870995017,23cyclictest15481-21idleruntime-cro08:50:121
1186999505,24cyclictest7240-21cat08:25:360
11869995023,13cyclictest7750irq/56-dwc_otg11:55:120
11870994924,11cyclictest149-21systemd-journal08:41:561
11870994921,12cyclictest22513-21latency_hist07:40:121
11869994924,4cyclictest15745-21grep12:15:300
1186999491,16cyclictest10988-21fschecks_count11:30:290
1187199481,44cyclictest7391-21ps08:25:392
1187199481,44cyclictest18458-21taskset07:25:412
1187199481,3cyclictest0-21swapper/207:10:362
11869994827,6cyclictest7950irq/56-dwc_otg_09:50:400
1187199477,37cyclictest15416-21sort07:20:132
1187199473,36cyclictest7950irq/56-dwc_otg_08:50:142
1187199471,3cyclictest0-21swapper/207:22:112
11870994721,11cyclictest13370-21sshd07:14:211
11870994716,13cyclictest0-21swapper/108:31:431
11869994713,16cyclictest22486-21munin-plugin-st07:40:120
11869994712,18cyclictest3339-21cut08:15:300
11870994622,12cyclictest17446-21latency_hist08:55:121
1186999451,27cyclictest12299-21/usr/sbin/munin07:10:300
1186999443,34cyclictest15801-21kworker/0:208:54:540
1186999442,7cyclictest20587-21unixbench_singl09:00:390
113462443,35sleep00-21swapper/007:07:020
1187299437,5cyclictest7850irq/56-dwc_otg_09:30:233
1186999432,25cyclictest0-21swapper/008:32:530
1186999431,33cyclictest0-21swapper/009:10:140
11869994121,12cyclictest101ktimersoftd/007:17:120
125632400,2sleep337-21rcuc/308:40:323
1187299404,4cyclictest0-21swapper/309:55:263
11870994010,10cyclictest7850irq/56-dwc_otg_08:15:551
11869994017,13cyclictest0-21swapper/008:42:210
1187299391,17cyclictest7950irq/56-dwc_otg_09:10:233
1187299388,19cyclictest0-21swapper/311:01:413
1187299378,23cyclictest0-21swapper/307:20:243
1187299366,3cyclictest0-21swapper/310:28:143
11872993618,6cyclictest21167-21fschecks_count07:35:313
1187299361,24cyclictest31953-21latency11:55:323
1187299361,23cyclictest8700-21cut08:30:313
11872993611,14cyclictest0-21swapper/310:20:293
142932350,3sleep339-21ksoftirqd/311:34:523
1187299352,19cyclictest7950irq/56-dwc_otg_10:35:223
11872993515,5cyclictest16285-21ssh09:40:243
1187299345,14cyclictest7950irq/56-dwc_otg_08:15:303
1187299343,19cyclictest7950irq/56-dwc_otg_10:30:273
1187299343,15cyclictest7950irq/56-dwc_otg_08:20:413
11872993419,5cyclictest26649-21sh11:10:263
11872993419,5cyclictest26649-21sh11:10:263
130182330,5sleep31187299cyclictest07:10:403
1187299337,8cyclictest7950irq/56-dwc_otg_12:30:243
11872993329,2cyclictest0-21swapper/312:05:353
11872993327,4cyclictest5504-21latency_hist10:45:133
11872993327,4cyclictest5504-21latency_hist10:45:133
11872993325,5cyclictest24007-21rm09:49:273
11872993325,5cyclictest24007-21rm09:49:273
1187299331,4cyclictest16566-21ls12:15:403
1187299331,13cyclictest7750irq/56-dwc_otg09:50:333
11872993311,12cyclictest0-21swapper/312:15:143
11872993311,12cyclictest0-21swapper/312:15:143
9892321,2sleep30-21swapper/310:00:293
1187299322,14cyclictest7850irq/56-dwc_otg_10:17:533
11872993215,4cyclictest8908-21latency_hist10:10:133
1187299321,23cyclictest3172-21latency_hist09:25:123
1187299321,23cyclictest0-21swapper/308:50:193
1187299317,4cyclictest0-21swapper/311:20:413
1187299316,11cyclictest37-21rcuc/309:15:373
1187299316,11cyclictest37-21rcuc/309:15:373
1187299314,6cyclictest7850irq/56-dwc_otg_11:50:293
1187299313,3cyclictest7950irq/56-dwc_otg_12:35:243
1187299312,3cyclictest7950irq/56-dwc_otg_09:35:263
1187299312,3cyclictest0-21swapper/311:25:243
1187299311,6cyclictest22354-21grep09:05:413
1187299311,5cyclictest23111-21ssh11:45:253
11872993112,4cyclictest0-21swapper/311:44:123
11872993112,4cyclictest0-21swapper/311:44:123
1187299309,4cyclictest30630-21missed_timers08:00:333
1187299306,4cyclictest7850irq/56-dwc_otg_07:30:273
1187299305,6cyclictest7950irq/56-dwc_otg_09:00:283
1187299305,5cyclictest7950irq/56-dwc_otg_08:10:413
1187299305,4cyclictest7950irq/56-dwc_otg_10:12:313
1187299305,11cyclictest7950irq/56-dwc_otg_11:15:463
1187299303,13cyclictest7950irq/56-dwc_otg_12:00:313
11872993022,5cyclictest22544-21wget11:05:273
1187299296,13cyclictest37-21rcuc/308:45:423
1187299296,13cyclictest37-21rcuc/308:45:423
1187299283,6cyclictest13985-21fschecks_time07:15:303
1187299283,3cyclictest7750irq/56-dwc_otg12:25:373
1187299283,12cyclictest7950irq/56-dwc_otg_07:55:373
1187299282,13cyclictest7950irq/56-dwc_otg_10:50:343
107822284,11sleep110862-21/usr/sbin/munin07:05:371
319292270,2sleep331930-21wget08:05:263
319292270,2sleep331930-21wget08:05:263
1187299276,6cyclictest7950irq/56-dwc_otg_08:38:363
1187299276,10cyclictest4750irq/17-3f00b88007:50:403
1187299272,13cyclictest0-21swapper/307:47:413
1187299271,13cyclictest7950irq/56-dwc_otg_08:55:323
1187299264,3cyclictest7750irq/56-dwc_otg07:40:423
1187299261,2cyclictest0-21swapper/308:25:293
1187299261,15cyclictest0-21swapper/307:25:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional