You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 22:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun Mar 22, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2811921240,6sleep21683199cyclictest09:49:272
2690221210,2sleep10-21swapper/109:05:341
2281521130,16sleep01682999cyclictest11:40:250
2392121040,6sleep11683099cyclictest10:23:001
225162950,2sleep20-21swapper/212:19:482
1682999959,8cyclictest101ktimersoftd/010:05:180
111292870,1sleep00-21swapper/009:28:360
16829998323,8cyclictest0-21swapper/009:00:190
62232810,2sleep30-21swapper/311:20:233
285272800,5sleep01682999cyclictest11:46:450
250282770,2sleep20-21swapper/212:21:492
250282770,2sleep20-21swapper/212:21:492
16829997527,11cyclictest0-21swapper/009:10:180
16829997469,3cyclictest0-21swapper/012:32:090
16829997469,3cyclictest0-21swapper/012:32:090
130662730,2sleep10-21swapper/109:30:301
16829997130,10cyclictest0-21swapper/011:50:190
1682999706,38cyclictest0-21swapper/011:15:210
16829997015,30cyclictest0-21swapper/008:30:190
16829996946,10cyclictest101ktimersoftd/010:05:080
52882680,1sleep20-21swapper/211:19:292
288132680,2sleep10-21swapper/109:10:291
16829996817,26cyclictest7853-21latency_hist09:25:080
91852650,2sleep20-21swapper/209:25:322
1682999656,25cyclictest0-21swapper/008:45:230
16829996510,32cyclictest0-21swapper/010:15:350
1683299649,10cyclictest7950irq/56-dwc_otg_11:45:083
1682999649,12cyclictest0-21swapper/008:55:190
1682999649,12cyclictest0-21swapper/008:55:190
1682999642,48cyclictest32688-21latency_hist09:55:080
16829996315,29cyclictest15419-21/usr/sbin/munin11:30:360
1682999622,46cyclictest13-21rcuc/012:10:340
1683299616,4cyclictest0-21swapper/311:00:263
16831996118,27cyclictest7308-21ssh11:20:382
16831996117,27cyclictest9643-21/usr/sbin/munin10:05:332
1682999613,32cyclictest13-21rcuc/010:25:340
1682999613,32cyclictest13-21rcuc/010:25:340
16829996124,9cyclictest0-21swapper/007:40:200
1682999612,38cyclictest4338-21grep12:35:370
1682999611,8cyclictest7750irq/56-dwc_otg10:40:210
1683299606,4cyclictest7850irq/56-dwc_otg_10:45:283
1683299601,3cyclictest0-21swapper/310:17:093
16831996017,26cyclictest13615-21/usr/sbin/munin08:30:222
1682999606,39cyclictest25868-21latency_hist11:05:080
1682999606,2cyclictest0-21swapper/011:10:220
1682999602,31cyclictest6438-21fschecks_count08:10:240
16829996022,4cyclictest0-21swapper/010:10:190
16829996022,4cyclictest0-21swapper/010:10:190
16829996020,12cyclictest0-21swapper/009:30:200
1683299596,10cyclictest0-21swapper/310:40:353
16831995919,21cyclictest23857-21ssh11:40:462
16831995918,24cyclictest1681-21ssh12:33:102
16831995918,24cyclictest1681-21ssh12:33:102
1682999597,38cyclictest13-21rcuc/010:58:550
1682999597,38cyclictest13-21rcuc/010:58:550
1682999594,31cyclictest25892-21kworker/u8:309:45:190
1682999593,32cyclictest23377-21unixbench_multi07:25:380
16832995817,38cyclictest0-21swapper/307:40:223
16831995818,24cyclictest29538-21ssh11:09:412
16829995833,9cyclictest101ktimersoftd/009:35:310
16829995825,31cyclictest101ktimersoftd/008:15:190
16829995813,25cyclictest22184-21latency_hist07:25:080
1682999581,29cyclictest21209-21kworker/0:210:22:380
1682999581,29cyclictest18482-21latency_hist07:15:080
1683299572,3cyclictest0-21swapper/311:48:063
1683299571,5cyclictest7950irq/56-dwc_otg_10:05:183
1683299571,4cyclictest6512-21/usr/sbin/munin12:00:333
1683299571,19cyclictest0-21swapper/308:07:033
16831995715,15cyclictest7750irq/56-dwc_otg09:05:262
1683299561,6cyclictest29265-21ssh10:30:083
1683299561,6cyclictest29265-21ssh10:30:083
1683299561,6cyclictest26459-21tune2fs09:05:273
1683299561,5cyclictest7976-21/usr/sbin/munin08:15:293
1683299561,5cyclictest4137-21ssh12:35:353
1683299561,10cyclictest7850irq/56-dwc_otg_10:20:213
16829995629,12cyclictest7950irq/56-dwc_otg_12:20:300
16829995629,12cyclictest7950irq/56-dwc_otg_12:20:300
1682999562,19cyclictest7950irq/56-dwc_otg_11:20:200
1682999561,32cyclictest14875-21sed10:50:360
1682999561,30cyclictest1767-21ssh10:35:280
16832995513,4cyclictest21681-21latency_hist11:00:093
16832995513,4cyclictest21681-21latency_hist11:00:093
16832995511,7cyclictest25217-21unixbench_singl07:30:353
1683299551,10cyclictest2329-21if_wlan011:15:273
16831995516,24cyclictest15140-21ssh09:33:242
1682999551,30cyclictest29584-21wget07:45:240
1682999551,30cyclictest29584-21wget07:45:240
1683299541,2cyclictest0-21swapper/312:05:293
1683299541,2cyclictest0-21swapper/310:30:313
16831995416,20cyclictest16138-21sh12:10:442
16830995423,5cyclictest23-21ksoftirqd/109:50:191
16829995437,4cyclictest7850irq/56-dwc_otg_11:25:220
16829995430,14cyclictest101ktimersoftd/012:05:080
1682999542,33cyclictest9-21ksoftirqd/008:35:320
1682999542,30cyclictest1757-21unixbench_multi07:55:360
1682999541,29cyclictest0-21swapper/012:25:220
1683299536,4cyclictest0-21swapper/309:31:563
1683299536,44cyclictest16849-21ls10:15:093
1683299536,44cyclictest16849-21ls10:15:093
1683299531,8cyclictest1814-21sh11:54:423
1683299531,4cyclictest27061-21timerandwakeup11:05:343
1683299531,3cyclictest0-21swapper/310:51:023
1683199532,11cyclictest7750irq/56-dwc_otg09:55:192
16831995319,12cyclictest30959-21df11:50:242
16831995314,12cyclictest3549-21/usr/sbin/munin12:35:272
1682999534,7cyclictest0-21swapper/008:40:200
16829995324,8cyclictest20239-21cron07:20:070
1683299528,4cyclictest0-21swapper/309:45:503
1683299521,9cyclictest0-21swapper/308:35:443
16831995217,21cyclictest12257-21nfs4_client08:25:292
16831995216,20cyclictest1145-21rm11:14:312
16831995214,24cyclictest17673-21sh10:55:112
16831995214,24cyclictest17673-21sh10:55:112
16831995212,11cyclictest7850irq/56-dwc_otg_10:40:222
1682999523,28cyclictest6203-21idleruntime-cro12:00:080
1682999523,28cyclictest6203-21idleruntime-cro12:00:080
16829995223,8cyclictest385-21sh07:55:070
16829995220,16cyclictest0-21swapper/010:45:230
79242510,1sleep20-21swapper/210:04:172
1683299515,3cyclictest7950irq/56-dwc_otg_09:24:523
1683299515,3cyclictest7750irq/56-dwc_otg10:37:283
1683299511,4cyclictest7950irq/56-dwc_otg_08:30:323
1683299511,3cyclictest0-21swapper/311:25:443
1683299511,10cyclictest7950irq/56-dwc_otg_09:42:333
1683299511,10cyclictest7950irq/56-dwc_otg_09:42:333
16831995113,12cyclictest917-21snmpd10:34:172
1682999514,24cyclictest20693-21ssh09:40:210
1682999514,24cyclictest20693-21ssh09:40:210
1682999512,32cyclictest9-21ksoftirqd/011:37:260
1682999511,19cyclictest0-21swapper/011:05:340
1683299503,3cyclictest0-21swapper/312:27:543
1683299502,9cyclictest0-21swapper/307:46:373
1683299502,9cyclictest0-21swapper/307:46:373
1683299501,5cyclictest0-21swapper/311:15:013
16832995015,5cyclictest7989-21sh10:04:243
1683299501,3cyclictest7750irq/56-dwc_otg12:24:323
1683299501,3cyclictest7750irq/56-dwc_otg12:24:323
1683199508,22cyclictest7950irq/56-dwc_otg_09:39:142
16831995020,18cyclictest31173-21date12:30:082
16831995018,17cyclictest7750irq/56-dwc_otg09:11:192
16831995017,19cyclictest19304-21ssh10:17:192
16831995017,13cyclictest11910-21unixbench_multi12:05:362
16829995025,6cyclictest0-21swapper/009:05:210
16829995023,10cyclictest101ktimersoftd/012:05:290
1683299491,5cyclictest0-21swapper/309:25:233
1683299491,3cyclictest7750irq/56-dwc_otg12:13:343
1683299491,3cyclictest7750irq/56-dwc_otg09:55:193
1683299491,12cyclictest7950irq/56-dwc_otg_09:03:063
1683199493,12cyclictest2407-21rm11:55:202
1683199493,12cyclictest2407-21rm11:55:202
16831994919,20cyclictest1530-21ssh09:16:252
16831994919,12cyclictest20852-21apt-get10:20:092
16831994918,14cyclictest7950irq/56-dwc_otg_11:46:042
16830994918,15cyclictest21546-21switchtime09:40:331
16830994918,15cyclictest21546-21switchtime09:40:331
1682999494,5cyclictest0-21swapper/010:30:190
1683299482,5cyclictest0-21swapper/308:28:113
1683299482,3cyclictest7750irq/56-dwc_otg09:12:163
1683299481,4cyclictest17317-21processes09:35:323
16831994813,11cyclictest7850irq/56-dwc_otg_11:35:082
1683099481,8cyclictest0-21swapper/110:40:191
1683299471,4cyclictest25883-21apt-get07:35:193
1683299471,3cyclictest0-21swapper/311:38:233
16831994719,13cyclictest22660-21ssh09:41:592
16831994719,13cyclictest22660-21ssh09:41:592
16831994715,11cyclictest2165-21sshd07:59:342
16831994713,21cyclictest7850irq/56-dwc_otg_11:26:402
16831994710,14cyclictest25336-21taskset07:31:202
16831994710,10cyclictest0-21swapper/210:46:292
1682999472,23cyclictest11880-21tune2fs08:25:240
1682999471,39cyclictest0-21swapper/007:35:330
16829994712,16cyclictest908-21runrttasks09:56:480
1683299464,5cyclictest27713-21diskmemload11:31:243
1683299463,3cyclictest7950irq/56-dwc_otg_12:33:433
1683299463,3cyclictest7950irq/56-dwc_otg_12:33:433
1683299463,10cyclictest7950irq/56-dwc_otg_07:26:213
1683299461,4cyclictest503-21in:imuxsock12:19:413
1683199467,25cyclictest0-21swapper/208:35:232
1683199467,24cyclictest0-21swapper/207:10:222
16831994630,13cyclictest6180-21apt-get08:10:192
16831994616,19cyclictest24153-21date09:00:082
16831994616,19cyclictest24153-21date09:00:082
16831994613,20cyclictest7497-21chrt09:24:442
16831994611,14cyclictest16036-21taskset10:13:572
16831994611,14cyclictest16036-21taskset10:13:572
1682999461,24cyclictest19651-21tr12:15:310
1683299453,7cyclictest0-21swapper/308:20:183
1683299451,3cyclictest7750irq/56-dwc_otg09:51:593
16831994519,18cyclictest3497-21ssh10:37:192
16831994515,18cyclictest7850irq/56-dwc_otg_10:52:192
16831994511,14cyclictest3767-21taskset08:01:532
16831994510,27cyclictest21718-21chrt07:21:142
1683099457,23cyclictest0-21swapper/108:10:201
1683299442,4cyclictest0-21swapper/309:17:363
1683299441,5cyclictest5573-21ssh11:59:003
1683299441,5cyclictest5573-21ssh11:59:003
16831994420,16cyclictest8857-21ssh12:02:272
16831994412,17cyclictest7850irq/56-dwc_otg_11:35:082
16831994412,13cyclictest30669-21ssh09:51:382
16831994411,13cyclictest28362-21ssh10:28:332
16831994411,13cyclictest28362-21ssh10:28:332
16831994410,23cyclictest23199-21ssh11:00:382
1683099441,6cyclictest25620-21ssh10:25:221
1683099441,6cyclictest25620-21ssh10:25:221
1682999446,30cyclictest0-21swapper/009:18:320
16829994413,21cyclictest0-21swapper/008:22:540
1683299437,24cyclictest0-21swapper/307:55:213
16831994318,11cyclictest9294-21cron08:17:082
1683099431,7cyclictest0-21swapper/107:30:191
16829994315,16cyclictest0-21swapper/008:05:080
164232432,35sleep00-21swapper/007:08:240
1683199429,13cyclictest25713-21taskset09:03:002
1683199429,13cyclictest11433-21latency_hist08:25:082
16831994211,17cyclictest7950irq/56-dwc_otg_08:50:352
1683099422,4cyclictest0-21swapper/111:25:221
16831994112,11cyclictest0-21swapper/207:26:352
1683099413,5cyclictest0-21swapper/112:15:201
1683099412,7cyclictest0-21swapper/112:35:221
16830994111,21cyclictest7850irq/56-dwc_otg_09:00:191
16829994119,12cyclictest101ktimersoftd/007:31:120
1683199403,6cyclictest7850irq/56-dwc_otg_07:45:222
1683199403,6cyclictest7850irq/56-dwc_otg_07:45:222
1683199402,31cyclictest0-21swapper/208:05:142
16831994018,7cyclictest31749-21irqstats07:50:282
1683099405,7cyclictest7850irq/56-dwc_otg_11:55:361
1683099405,7cyclictest7850irq/56-dwc_otg_11:55:361
16830994020,5cyclictest24044-21switchtime12:20:331
16830994020,5cyclictest24044-21switchtime12:20:331
1683099394,4cyclictest7850irq/56-dwc_otg_08:45:201
16830993915,5cyclictest0-21swapper/108:15:361
1683199388,10cyclictest7850irq/56-dwc_otg_07:18:462
1683199386,22cyclictest0-21swapper/207:44:212
16831993826,9cyclictest17118-21apt-get08:40:202
1683099388,7cyclictest7750irq/56-dwc_otg09:45:361
1683099388,20cyclictest0-21swapper/112:30:191
1683099388,20cyclictest0-21swapper/112:30:191
1683099383,9cyclictest0-21swapper/109:55:171
1683099383,7cyclictest0-21swapper/111:10:201
16830993819,12cyclictest221ktimersoftd/110:30:211
16830993815,5cyclictest31905-21grep11:50:361
1683099381,30cyclictest1519-21grep10:35:251
1683099381,23cyclictest22117-21latency_hist11:40:081
16829993825,3cyclictest0-21swapper/008:50:370
1682999381,20cyclictest2500-21wget08:00:210
1683299377,4cyclictest0-21swapper/308:55:213
1683299377,4cyclictest0-21swapper/308:55:213
16831993728,7cyclictest18926-21apt-get08:45:222
1683099377,7cyclictest7850irq/56-dwc_otg_08:05:081
1683099374,4cyclictest0-21swapper/107:20:241
16830993719,5cyclictest7950irq/56-dwc_otg_10:00:261
16830993718,7cyclictest0-21swapper/112:10:211
16830993714,4cyclictest7750irq/56-dwc_otg11:40:361
16830993710,3cyclictest0-21swapper/107:50:201
1683199368,8cyclictest0-21swapper/207:35:272
1683099369,4cyclictest0-21swapper/111:45:301
1683099365,6cyclictest7850irq/56-dwc_otg_09:15:351
1683099365,14cyclictest7850irq/56-dwc_otg_09:35:221
1683099363,7cyclictest7950irq/56-dwc_otg_07:15:201
16830993619,5cyclictest8721-21apt10:05:181
16830993613,6cyclictest23663-21grep08:55:351
16830993613,6cyclictest23663-21grep08:55:351
16830993612,4cyclictest0-21swapper/109:25:321
102212360,2sleep10-21swapper/108:20:261
1683299356,14cyclictest0-21swapper/308:00:213
1683099356,13cyclictest7950irq/56-dwc_otg_07:50:091
1683099356,13cyclictest7950irq/56-dwc_otg_07:50:091
1683099355,13cyclictest7850irq/56-dwc_otg_12:05:341
1683099352,15cyclictest7850irq/56-dwc_otg_08:25:211
1683099351,6cyclictest7950irq/56-dwc_otg_11:06:251
16830993516,5cyclictest908-21runrttasks10:16:181
1683099351,3cyclictest7950irq/56-dwc_otg_07:10:201
32022340,3sleep10-21swapper/111:15:421
1683299347,14cyclictest0-21swapper/307:50:223
1683099347,12cyclictest7950irq/56-dwc_otg_11:20:191
1683099343,7cyclictest970-21fschecks_time07:55:261
1683099341,6cyclictest22521-21grep07:25:261
1683099341,16cyclictest1341050ssh10:10:271
1683099341,16cyclictest1341050ssh10:10:271
1683099334,7cyclictest7850irq/56-dwc_otg_10:50:251
1683099334,6cyclictest7850irq/56-dwc_otg_07:35:231
1683099333,5cyclictest7950irq/56-dwc_otg_12:00:181
16830993315,4cyclictest27713-21diskmemload12:25:591
16830993312,4cyclictest20719-21/usr/sbin/munin08:50:311
16830993312,4cyclictest0-21swapper/111:00:351
16830993310,5cyclictest0-21swapper/110:58:481
16830993310,5cyclictest0-21swapper/110:58:481
16830993216,5cyclictest11889-1kworker/u9:211:30:221
16830993210,5cyclictest7750irq/56-dwc_otg09:20:321
1683099312,12cyclictest0-21swapper/108:40:561
1683099311,10cyclictest9816-21fschecks_count10:45:281
1683099306,11cyclictest13642-21fschecks_count08:30:241
16832992919,7cyclictest20967-21fschecks_count08:50:263
1683099294,12cyclictest7850irq/56-dwc_otg_08:37:581
1683299286,6cyclictest7950irq/56-dwc_otg_08:40:343
1683299282,3cyclictest0-21swapper/307:15:343
1683299281,21cyclictest0-21swapper/307:20:393
1683099287,10cyclictest28158-21kernelversion07:40:281
1683099281,6cyclictest5511-21unixbench_singl08:05:371
163572284,14sleep10-21swapper/107:07:321
154672285,14sleep215552-21cat07:05:262
16832992718,6cyclictest18520-21latency_hist07:15:083
1683299261,15cyclictest0-21swapper/308:10:283
1683299261,12cyclictest7950irq/56-dwc_otg_08:48:113
162322263,15sleep30-21swapper/307:06:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional