You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 12:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Tue Dec 02, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
566821020,4sleep35669-21threads22:20:393
804521010,2sleep20-21swapper/222:24:572
408899913,82cyclictest7950irq/56-dwc_otg_23:35:240
4088999120,42cyclictest8249-21latency_hist22:25:120
4088999016,45cyclictest0-21swapper/019:55:110
4088998921,55cyclictest13937-21missed_timers21:05:330
4088998620,50cyclictest16461-21unixbench_multi21:10:400
4088998533,15cyclictest14752-21ssh23:15:230
4088998524,39cyclictest2894-21ls23:40:400
4088998519,44cyclictest8667-21cat19:20:380
4088998519,44cyclictest8667-21cat19:20:380
4088998443,7cyclictest5927-21apt-get19:15:230
4088998424,33cyclictest13297-21sed21:05:120
4088998225,45cyclictest8602-21sh23:49:050
4088998121,12cyclictest2495-21aten_rbpower_po20:35:260
4088998119,38cyclictest3764-21cat00:25:330
4088998117,54cyclictest27566-21ssh22:50:070
4088998115,49cyclictest0-21swapper/000:05:290
4088998025,33cyclictest10473-21/usr/sbin/munin00:35:380
4088998023,46cyclictest20610-21ssh21:16:050
4088998020,52cyclictest28759-21grep20:15:410
67822790,2sleep36770-21apt-get00:30:233
4088997927,28cyclictest10580-21munin-run23:10:110
4088997926,37cyclictest0-21swapper/021:35:230
4088997922,36cyclictest8242-21ssh00:31:020
4088997921,48cyclictest17918-21ssh22:36:150
4088997920,47cyclictest20114-21ssh23:21:030
4088997920,40cyclictest4026-21taskset22:19:380
4088997818,39cyclictest22504-21uniq20:00:310
4088997726,35cyclictest19939-21rm21:57:300
4088997726,35cyclictest19939-21rm21:57:300
4088997723,40cyclictest8914-21/usr/sbin/munin22:25:310
4088997720,43cyclictest28081-21unixbench_multi21:25:420
4088997624,35cyclictest2036-21ssh00:23:030
4088997621,38cyclictest28595-21/usr/sbin/munin22:50:330
4088997520,41cyclictest26799-21ssh22:05:460
4088997520,36cyclictest15414-21rm23:57:300
153162750,5sleep3409799cyclictest19:40:303
4088997420,37cyclictest0-21swapper/019:55:330
4088997419,38cyclictest14428-21modprobe23:15:110
4088997417,41cyclictest0-21swapper/022:10:280
4088997328,34cyclictest17011-21/usr/sbin/munin00:00:220
4088997328,27cyclictest4158-21aten_rbpower_cu19:10:250
4088997318,39cyclictest15180-21ssh22:33:310
4088997315,47cyclictest0-21swapper/000:15:390
4088997228,40cyclictest22649-21timerandwakeup22:00:390
4088997225,38cyclictest13215-21ssh21:49:340
4088997218,44cyclictest6944-21rm21:40:390
4088997218,44cyclictest25610-21awk00:10:430
4088997214,43cyclictest20210-21ssh22:40:230
4088997214,43cyclictest20210-21ssh22:40:230
257662720,5sleep2409699cyclictest22:46:312
408899716,49cyclictest9-21ksoftirqd/023:00:110
4088997125,33cyclictest5412-21seq20:40:450
4088997112,42cyclictest0-21swapper/021:20:250
4088997020,37cyclictest25621-21ssh23:29:230
4088997018,39cyclictest605-21rm21:32:490
4088996919,40cyclictest14903-21diskmemload23:32:400
4088996822,35cyclictest140-21systemd-journal20:06:170
4088996820,31cyclictest15732-21ssh21:51:530
4088996818,35cyclictest0-21swapper/019:50:110
4088996718,37cyclictest16285-21chrt19:41:520
4088996715,40cyclictest0-21swapper/020:27:180
4088996713,42cyclictest27046-21taskset20:11:140
4088996713,42cyclictest27046-21taskset20:11:140
4088996626,31cyclictest12907-21runrttasks23:55:080
4088996522,32cyclictest0-21swapper/020:30:250
4088996517,34cyclictest0-21swapper/023:04:120
408899651,54cyclictest9816-21aten_rbpower_po20:55:290
4088996514,39cyclictest9645-21cut19:25:280
4088996425,26cyclictest9411-21sshd20:54:250
4088996422,24cyclictest14663-21taskset19:39:120
108962640,5sleep3409799cyclictest23:10:233
4088996118,27cyclictest0-21swapper/019:31:150
4088996032,12cyclictest7950irq/56-dwc_otg_20:45:410
4088995912,31cyclictest0-21swapper/020:22:210
257302590,5sleep225717-21ssh22:05:292
41362580,2sleep20-21swapper/200:25:392
87742550,2sleep30-21swapper/321:43:363
4096995311,26cyclictest21559-21ssh22:00:192
409799519,34cyclictest7950irq/56-dwc_otg_20:35:253
67972500,2sleep20-21swapper/221:40:372
4096995012,27cyclictest27840-21sh22:50:132
263692500,5sleep37950irq/56-dwc_otg_20:10:333
263692500,5sleep37950irq/56-dwc_otg_20:10:333
4096994811,22cyclictest22752-21sed20:00:342
409799477,26cyclictest0-21swapper/319:25:273
409799475,12cyclictest37-21rcuc/322:00:123
409799475,12cyclictest37-21rcuc/322:00:123
409799465,26cyclictest0-21swapper/323:35:273
4096994625,8cyclictest17014-21apt-get00:00:232
4096994615,8cyclictest13339-21apt-get21:05:232
4093994621,6cyclictest17495-21latency_hist21:55:121
409399461,9cyclictest20746-21ssh00:05:221
279992460,3sleep227607-21/usr/sbin/munin20:15:312
409799457,25cyclictest0-21swapper/320:50:183
4096994520,5cyclictest26101-21fschecks_time20:10:302
4096994520,5cyclictest26101-21fschecks_time20:10:302
4096994520,4cyclictest18793-21idleruntime-cro21:15:112
4096994518,7cyclictest0-21swapper/222:00:122
4096994518,7cyclictest0-21swapper/222:00:122
4096994511,14cyclictest18517-21latency_hist19:50:122
4093994522,8cyclictest30585-21ssh00:17:271
409699448,22cyclictest3040-21/usr/sbin/munin23:40:422
4093994424,6cyclictest7750irq/56-dwc_otg23:15:251
409799436,25cyclictest0-21swapper/320:30:273
409799433,30cyclictest7750irq/56-dwc_otg22:50:263
4097994319,5cyclictest13313-21aten_rbpower_vo19:35:263
409699438,9cyclictest2687-21ssh22:16:572
4096994324,7cyclictest9762-21aten_rbpower_cu20:55:282
4096994312,17cyclictest20248-21ssh23:21:202
4097994220,4cyclictest39-21ksoftirqd/321:35:363
409699427,10cyclictest29596-21ssh21:28:372
409699426,15cyclictest0-21swapper/219:55:172
409699425,19cyclictest7750irq/56-dwc_otg19:15:262
409699423,23cyclictest24775-21ssh21:21:392
4096994227,5cyclictest28850-21ssh00:15:312
4096994224,7cyclictest7950irq/56-dwc_otg_22:41:552
4096994224,7cyclictest7950irq/56-dwc_otg_22:41:552
4096994216,4cyclictest7950irq/56-dwc_otg_22:25:392
4093994213,5cyclictest23-21ksoftirqd/122:50:281
409799417,25cyclictest0-21swapper/323:55:273
409799414,25cyclictest7850irq/56-dwc_otg_21:05:333
409799414,24cyclictest32347-21grep20:25:413
4097994119,13cyclictest39-21ksoftirqd/319:20:353
4097994119,13cyclictest39-21ksoftirqd/319:20:353
409699416,11cyclictest7850irq/56-dwc_otg_23:18:182
4096994117,13cyclictest7850irq/56-dwc_otg_23:29:482
4096994115,12cyclictest7950irq/56-dwc_otg_21:18:562
4096994113,16cyclictest0-21swapper/220:30:272
4096994113,12cyclictest17104-21ssh21:54:392
4096994111,11cyclictest26443-21ssh00:11:562
409399419,5cyclictest221ktimersoftd/123:35:021
4093994126,4cyclictest7850irq/56-dwc_otg_22:30:461
4093994114,15cyclictest221ktimersoftd/122:28:251
214552410,1sleep10-21swapper/121:17:451
200232410,3sleep337-21rcuc/300:04:183
4097994020,8cyclictest381ktimersoftd/322:42:113
4097994020,8cyclictest381ktimersoftd/322:42:113
409799401,7cyclictest39-21ksoftirqd/320:00:313
409699403,12cyclictest4413-21fschecks_time19:10:292
4096994023,8cyclictest7950irq/56-dwc_otg_23:00:262
4096994012,12cyclictest13297-21ssh00:39:512
4096994010,11cyclictest2740-21fschecks_time20:35:292
409399407,23cyclictest0-21swapper/100:10:171
4093994014,15cyclictest221ktimersoftd/100:34:121
4093994013,9cyclictest7750irq/56-dwc_otg20:00:371
409399401,18cyclictest0-21swapper/121:40:351
180222400,4sleep3381ktimersoftd/321:13:403
409799395,22cyclictest381ktimersoftd/321:29:473
409799395,19cyclictest39-21ksoftirqd/319:15:373
4097993921,4cyclictest7850irq/56-dwc_otg_22:00:323
409699399,4cyclictest8852-21ssh23:49:342
409699398,17cyclictest8312-21taskset00:31:102
409699397,14cyclictest7850irq/56-dwc_otg_23:07:272
409699397,14cyclictest15346-21sh22:33:532
409699397,13cyclictest7950irq/56-dwc_otg_00:06:242
409699395,20cyclictest7950irq/56-dwc_otg_21:35:262
409699395,13cyclictest31155-21sh22:11:452
409699394,13cyclictest525-21ssh21:32:392
409699393,10cyclictest0-21swapper/223:35:312
409699392,21cyclictest7850irq/56-dwc_otg_23:35:122
4096993914,4cyclictest16557-21sh19:45:112
4096993910,20cyclictest1895-21ssh00:22:442
409399398,8cyclictest221ktimersoftd/123:52:181
4093993922,9cyclictest221ktimersoftd/100:20:241
409399391,6cyclictest0-21swapper/121:01:271
4093993914,8cyclictest3171-21/usr/sbin/munin23:00:311
4093993913,17cyclictest0-21swapper/119:40:251
409799386,6cyclictest39-21ksoftirqd/321:05:123
409799385,13cyclictest39-21ksoftirqd/321:15:393
4097993825,5cyclictest7750irq/56-dwc_otg22:45:413
409799382,21cyclictest7750irq/56-dwc_otg22:36:163
4097993820,5cyclictest39-21ksoftirqd/319:55:243
409799381,27cyclictest26476-21ls22:05:383
409699387,16cyclictest0-21swapper/222:58:402
409699383,14cyclictest24719-21ntp_states20:05:392
409399389,18cyclictest221ktimersoftd/119:15:261
409399387,21cyclictest0-21swapper/119:20:251
409399387,21cyclictest0-21swapper/119:20:251
409399382,20cyclictest0-21swapper/121:45:311
4093993819,9cyclictest15264-21chrt21:10:241
4093993818,7cyclictest7750irq/56-dwc_otg00:25:301
4093993818,5cyclictest31945-21unixbench_singl21:30:401
4093993816,5cyclictest11111-21ls19:30:111
4093993816,11cyclictest7850irq/56-dwc_otg_22:58:421
4093993812,5cyclictest13994-21/usr/sbin/munin21:05:341
409399381,24cyclictest0-21swapper/122:42:101
409399381,24cyclictest0-21swapper/122:42:101
4097993730,4cyclictest1156-21sshd00:15:303
409799372,5cyclictest7850irq/56-dwc_otg_23:40:413
4097993716,5cyclictest12417-21sh23:54:033
409699374,11cyclictest13141-21ssh23:12:362
409399377,23cyclictest21-21rcuc/122:14:321
409399372,9cyclictest7950irq/56-dwc_otg_22:00:241
4093993721,9cyclictest30405-21ssh23:35:281
4093993720,8cyclictest7750irq/56-dwc_otg23:43:331
409399371,21cyclictest0-21swapper/120:50:181
409399371,18cyclictest21-21rcuc/122:22:301
409799369,5cyclictest39-21ksoftirqd/323:15:403
409799365,23cyclictest0-21swapper/323:03:153
409799363,8cyclictest11-21rcu_preempt22:27:553
4097993618,13cyclictest39-21ksoftirqd/322:55:263
409799360,29cyclictest0-21swapper/320:45:243
409699366,12cyclictest9146-21chrt19:23:222
409699366,12cyclictest9146-21chrt19:23:222
409699362,5cyclictest7950irq/56-dwc_otg_19:35:252
4096993610,13cyclictest11546-21rm23:52:152
409399368,6cyclictest21-21rcuc/120:05:331
409399366,16cyclictest21-21rcuc/123:27:001
409399366,16cyclictest21-21rcuc/122:47:201
409399364,20cyclictest221ktimersoftd/119:55:381
409399363,18cyclictest7850irq/56-dwc_otg_21:38:111
4093993624,4cyclictest7750irq/56-dwc_otg23:21:001
4093993616,6cyclictest7750irq/56-dwc_otg00:00:271
409399361,20cyclictest221ktimersoftd/122:08:471
4093993610,10cyclictest6552-21cat23:45:371
409799357,5cyclictest7750irq/56-dwc_otg23:25:253
409799357,20cyclictest7850irq/56-dwc_otg_22:35:113
409799355,20cyclictest0-21swapper/321:55:113
409799354,7cyclictest7750irq/56-dwc_otg19:30:403
4097993516,5cyclictest37-21rcuc/321:21:273
4097993514,7cyclictest37-21rcuc/300:22:363
409799351,24cyclictest24244-21apt-config00:10:123
4097993510,5cyclictest39-21ksoftirqd/321:35:113
4096993510,16cyclictest11836-21ssh21:46:462
409399357,17cyclictest221ktimersoftd/120:35:301
409399357,12cyclictest221ktimersoftd/120:30:251
409399352,21cyclictest10873-21/usr/sbin/munin23:10:221
409399352,19cyclictest7750irq/56-dwc_otg21:55:351
409399352,19cyclictest7750irq/56-dwc_otg21:55:351
409399351,16cyclictest1120-21runrttasks20:16:371
4093993510,15cyclictest7950irq/56-dwc_otg_22:20:041
409399350,22cyclictest4314-21cpuspeed19:10:261
409799346,6cyclictest7750irq/56-dwc_otg21:50:123
409799342,24cyclictest7750irq/56-dwc_otg23:20:593
409799341,24cyclictest7750irq/56-dwc_otg00:25:293
409699344,15cyclictest7850irq/56-dwc_otg_23:57:392
409699341,9cyclictest7950irq/56-dwc_otg_20:24:472
409699341,10cyclictest0-21swapper/220:41:132
409399346,12cyclictest21-21rcuc/121:24:231
409399344,17cyclictest0-21swapper/123:10:081
4093993422,4cyclictest7850irq/56-dwc_otg_22:36:181
409399342,20cyclictest0-21swapper/120:59:151
4093993419,8cyclictest7750irq/56-dwc_otg19:46:111
409399341,17cyclictest31017-21grep20:25:111
4088993420,11cyclictest7950irq/56-dwc_otg_19:10:120
288242340,5sleep1409399cyclictest21:27:031
409799334,19cyclictest7750irq/56-dwc_otg20:25:113
4097993315,7cyclictest7750irq/56-dwc_otg19:10:203
409799331,4cyclictest0-21swapper/323:45:393
4097993313,5cyclictest27620-21apt-get20:15:243
4097993312,10cyclictest39-21ksoftirqd/320:05:273
409799331,15cyclictest39-21ksoftirqd/319:55:113
4097993311,13cyclictest7750irq/56-dwc_otg20:43:473
409799331,10cyclictest31611-21ssh22:12:413
409699332,23cyclictest7950irq/56-dwc_otg_19:50:232
409699331,10cyclictest11520-21/usr/sbin/munin21:00:322
409399338,13cyclictest0-21swapper/100:35:281
409399332,25cyclictest7750irq/56-dwc_otg23:55:341
409799329,4cyclictest0-21swapper/300:15:023
409799326,5cyclictest7950irq/56-dwc_otg_23:30:313
4097993225,5cyclictest4323-21sh22:20:113
409799322,21cyclictest37-21rcuc/323:09:493
409799321,20cyclictest7750irq/56-dwc_otg19:50:113
409699329,4cyclictest7850irq/56-dwc_otg_19:26:032
409699326,13cyclictest7850irq/56-dwc_otg_20:45:412
409699323,16cyclictest7850irq/56-dwc_otg_22:36:242
409399323,20cyclictest0-21swapper/119:35:231
4097993111,10cyclictest381ktimersoftd/300:36:423
4096993113,9cyclictest11917-21/usr/sbin/munin19:30:342
409399316,16cyclictest31249-21sshd20:25:251
409399313,20cyclictest0-21swapper/119:33:271
409399311,8cyclictest6819-21processes20:45:381
4093993112,11cyclictest18631-21basename19:50:261
36042315,18sleep10-21swapper/119:07:231
409799303,17cyclictest39-21ksoftirqd/321:00:123
4096993014,12cyclictest31335-21aten_rbpower_po20:25:282
409399306,12cyclictest221ktimersoftd/120:11:391
409399306,12cyclictest221ktimersoftd/120:11:391
409399301,20cyclictest0-21swapper/120:40:231
4096992915,11cyclictest8334-21if_eth020:50:312
37602273,14sleep30-21swapper/319:09:233
34842274,15sleep20-21swapper/219:05:542
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional