You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sat Jul 12, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16396991383,97cyclictest24980-21ssh23:55:150
2954021240,3sleep3381ktimersoftd/319:45:233
1781221000,5sleep21639899cyclictest21:39:302
1639699911,80cyclictest0-21swapper/023:25:310
1639699892,70cyclictest0-21swapper/021:35:330
157092880,2sleep30-21swapper/323:42:223
1639699871,80cyclictest27381-21diskmemload23:20:160
1639699871,73cyclictest16541-21/usr/sbin/munin19:10:170
1639699871,73cyclictest0-21swapper/000:25:320
1639699871,6cyclictest28030-21fschecks_time21:10:210
1639699862,74cyclictest0-21swapper/000:39:220
1639699852,71cyclictest0-21swapper/019:50:470
1639699845,27cyclictest0-21swapper/019:35:150
1639699841,4cyclictest6997-21timerwakeupswit20:10:300
1639699831,70cyclictest0-21swapper/023:33:590
1639699831,67cyclictest31941-21fschecks_count21:15:210
1639699831,4cyclictest10578-21munin-run00:20:030
44712820,2sleep00-21swapper/022:04:120
1639699822,69cyclictest0-21swapper/022:58:470
1639699822,66cyclictest0-21swapper/023:14:540
1639699822,66cyclictest0-21swapper/020:50:310
1639699822,65cyclictest0-21swapper/019:26:230
1639699821,70cyclictest0-21swapper/000:00:430
1639699812,67cyclictest0-21swapper/022:15:560
1639699811,66cyclictest0-21swapper/023:44:470
1639699811,66cyclictest0-21swapper/021:09:570
1639699802,62cyclictest0-21swapper/020:30:130
1639699802,62cyclictest0-21swapper/020:30:130
1639699802,4cyclictest3515-21/usr/sbin/munin22:45:220
1639699801,66cyclictest0-21swapper/022:05:170
1639699791,6cyclictest29314-21aten_rbpower_cu19:45:200
1639699791,68cyclictest0-21swapper/000:31:050
281712780,4sleep028135-21ssh22:35:040
27532780,2sleep30-21swapper/322:00:413
1639699782,66cyclictest0-21swapper/000:20:260
1639699782,65cyclictest0-21swapper/022:39:570
1639699781,65cyclictest0-21swapper/023:47:090
1639699781,65cyclictest0-21swapper/023:47:090
1639699778,52cyclictest32444-21grep22:40:230
1639699778,52cyclictest32444-21grep22:40:230
1639699773,51cyclictest0-21swapper/023:15:170
1639699771,65cyclictest0-21swapper/023:38:580
1639699771,38cyclictest0-21swapper/021:35:030
16396997710,51cyclictest239092chrt22:29:360
1639699762,61cyclictest0-21swapper/020:38:470
1639699761,63cyclictest0-21swapper/021:24:050
1639699751,55cyclictest0-21swapper/020:19:150
1639699741,5cyclictest20336-21latency_hist20:50:030
1639699741,4cyclictest0-21swapper/023:52:300
1639699741,3cyclictest0-21swapper/021:43:440
1639699741,3cyclictest0-21swapper/021:43:440
16398997345,8cyclictest32141-21apt-get22:40:172
16398997345,8cyclictest32141-21apt-get22:40:172
1639699734,54cyclictest0-21swapper/000:14:270
1639699732,56cyclictest0-21swapper/023:04:310
1639699731,60cyclictest1005-21sh21:59:590
1639699731,55cyclictest0-21swapper/022:10:380
1639699728,53cyclictest10103-21ssh22:53:300
1639699722,6cyclictest0-21swapper/020:24:310
1639699722,59cyclictest0-21swapper/023:06:540
1639699721,4cyclictest18518-21fschecks_time19:15:220
1639699721,4cyclictest0-21swapper/022:24:130
1639699721,3cyclictest0-21swapper/000:06:340
1639699712,50cyclictest0-21swapper/019:34:480
1639699711,58cyclictest22242-21apt-get21:45:140
1639699711,56cyclictest0-21swapper/021:26:470
16398997027,25cyclictest1889-21ssh00:06:372
1639699701,52cyclictest0-21swapper/021:52:050
16398996934,13cyclictest9650-21latency22:10:252
1639799696,5cyclictest0-21swapper/120:05:191
1639699692,54cyclictest0-21swapper/020:41:360
1639699691,3cyclictest54092chrt20:07:380
1639699682,48cyclictest0-21swapper/020:27:410
16398996726,29cyclictest29817-21sed19:45:262
1639699671,57cyclictest0-21swapper/019:40:180
1639699671,57cyclictest0-21swapper/019:40:180
1639699671,4cyclictest0-21swapper/019:56:220
16398996631,13cyclictest14188-21timerwakeupswit20:30:302
16398996631,13cyclictest14188-21timerwakeupswit20:30:302
16398996530,22cyclictest5889-21missed_timers22:05:252
1639699652,49cyclictest0-21swapper/020:55:500
1640099646,15cyclictest12933-21ssh22:15:173
1640099644,16cyclictest7950irq/56-dwc_otg_21:00:173
16398996432,19cyclictest18176-21seq23:04:142
16398996428,12cyclictest15367-21ssh23:41:412
16398996427,16cyclictest7032-21sed20:10:312
16398996424,9cyclictest6242-21perl23:30:212
1639799644,14cyclictest8050irq/56-dwc_otg_22:45:321
1639799642,7cyclictest26286-21idleruntime-cro23:15:021
1639699641,4cyclictest3563-21taskset20:02:030
16398996341,7cyclictest19580-21ssh23:47:292
16398996341,7cyclictest19580-21ssh23:47:292
16398996330,12cyclictest30551-21ssh23:20:142
16398996327,21cyclictest19736-21ssh22:23:532
16398996327,12cyclictest23417-21ssh23:52:302
16398996325,16cyclictest30243-21grep23:20:032
16398996324,13cyclictest30424-21ssh22:37:142
1639699631,3cyclictest0-21swapper/021:01:490
152982630,5sleep20-21swapper/219:05:272
16398996233,12cyclictest3690-21ssh22:02:352
16398996232,17cyclictest16904-21ssh00:28:182
16398996230,13cyclictest3732-21sort22:45:192
16398996230,13cyclictest20390-21ssh00:34:112
16398996230,11cyclictest14973-21/usr/sbin/munin20:35:292
16398996229,11cyclictest24884-21ssh23:12:132
16398996228,22cyclictest24069-21ssh21:46:342
16398996226,11cyclictest31691-21ssh00:03:502
16398996223,20cyclictest11607-21/usr/sbin/munin00:20:252
1639699621,4cyclictest0-21swapper/019:24:010
16398996131,17cyclictest22798-21ssh00:37:012
16398996131,12cyclictest11542-21ssh23:36:402
16398996131,11cyclictest565-21ssh21:15:392
16398996130,19cyclictest6732-21ssh00:14:292
16398996129,11cyclictest27940-21ssh21:10:202
16398996128,19cyclictest22433-21ssh22:26:322
16398996127,23cyclictest27372-21latency_hist21:10:042
16398996127,13cyclictest32712-21ssh21:58:322
16398996127,12cyclictest10153-21ssh21:29:322
16398996126,22cyclictest8748-21ssh22:50:422
16398996126,14cyclictest25221-21ssh23:55:202
16398996126,12cyclictest26571-21ssh22:32:152
16398996031,21cyclictest22450-21ssh23:10:032
16398996030,13cyclictest12811-21ssh22:56:082
16398996028,18cyclictest13292-21ssh21:33:072
16398996028,12cyclictest4802-21ssh23:28:352
16398996027,14cyclictest16659-21fschecks_count19:10:192
16398996025,21cyclictest23444-21seq20:56:122
1639799602,4cyclictest314-21grep22:40:271
1639799602,4cyclictest314-21grep22:40:271
16397996014,4cyclictest0-21swapper/122:00:231
160932603,50sleep30-21swapper/319:09:383
16398995930,7cyclictest27975-21ssh21:51:452
16398995928,11cyclictest21099-21ntp_states20:50:272
16398995926,12cyclictest19736-21ssh21:40:362
16398995926,12cyclictest19736-21ssh21:40:362
1639799597,4cyclictest0-21swapper/121:49:431
1639799595,9cyclictest0-21swapper/100:30:171
1639799591,6cyclictest0-21swapper/100:00:221
16398995822,10cyclictest0-21swapper/219:50:222
1639799584,11cyclictest8050irq/56-dwc_otg_21:41:111
1639799584,11cyclictest8050irq/56-dwc_otg_21:41:111
1639799583,6cyclictest26858-21sh23:15:211
1639799582,6cyclictest7850irq/56-dwc_otg22:05:261
1639799581,8cyclictest12079-21grep00:20:321
1639799581,5cyclictest11140-21idleruntime-cro20:25:031
16398995734,8cyclictest27680-21df19:40:222
16398995734,8cyclictest27680-21df19:40:222
16398995727,11cyclictest6280-21ssh21:24:242
1639799578,4cyclictest7850irq/56-dwc_otg23:55:251
1639799574,4cyclictest30028-21proc_pri19:45:311
1639799572,6cyclictest28140-21latency_hist22:35:041
1639799571,7cyclictest8050irq/56-dwc_otg_00:15:031
192212560,3sleep119224-21fschecks_time23:05:201
16398995642,11cyclictest9958-21/usr/sbin/munin20:20:242
16398995624,13cyclictest2068-21latency_hist20:00:032
16398995622,14cyclictest144-21systemd-journal19:35:032
16398995622,11cyclictest15718-21sh22:18:352
1639799561,6cyclictest7850irq/56-dwc_otg23:20:231
16398995523,11cyclictest8050irq/56-dwc_otg_20:05:192
1639799554,6cyclictest21189-21ssh23:50:181
1639799554,6cyclictest16842-21/usr/sbin/munin19:10:211
1639799551,5cyclictest27317-21gpgconf19:40:031
1639799551,14cyclictest7950irq/56-dwc_otg_21:35:031
16398995425,10cyclictest249552sleep219:31:502
16398995423,13cyclictest18066-21chrt20:41:332
1639799549,4cyclictest0-21swapper/123:31:161
1639799545,6cyclictest8050irq/56-dwc_otg_20:50:171
1639799543,6cyclictest8050irq/56-dwc_otg_22:55:231
1639799541,6cyclictest144-21systemd-journal21:51:071
1639799541,6cyclictest0-21swapper/122:52:071
1639799541,13cyclictest8050irq/56-dwc_otg_00:09:321
16398995325,18cyclictest19457-21chrt19:16:002
16398995324,12cyclictest10547-21ssh00:20:012
1639799535,4cyclictest0-21swapper/123:44:261
1639799532,5cyclictest0-21swapper/121:35:581
1639799532,11cyclictest0-21swapper/120:42:031
16397995311,5cyclictest4857-21taskset23:28:421
16398995219,10cyclictest8050irq/56-dwc_otg_21:00:322
1639799526,9cyclictest7850irq/56-dwc_otg21:28:501
16397995214,9cyclictest8050irq/56-dwc_otg_21:55:201
1640099511,12cyclictest22768-21taskset23:10:173
1639799516,5cyclictest7950irq/56-dwc_otg_22:10:171
1639799516,42cyclictest20370-21cut19:20:191
1639799515,10cyclictest8050irq/56-dwc_otg_21:24:341
1639799513,13cyclictest0-21swapper/121:18:401
1639799512,12cyclictest7950irq/56-dwc_otg_23:04:431
1639799504,4cyclictest7850irq/56-dwc_otg20:55:121
1639799502,7cyclictest0-21swapper/120:30:391
1639799502,7cyclictest0-21swapper/120:30:391
1639799501,6cyclictest15078-21memory00:25:261
16398994925,17cyclictest21500-21taskset19:23:462
16398994823,12cyclictest7950irq/56-dwc_otg_20:04:582
1639799482,5cyclictest9996-21uniq23:35:251
157792483,36sleep00-21swapper/019:05:380
1640099477,25cyclictest0-21swapper/323:45:163
1640099477,25cyclictest0-21swapper/323:45:163
16398994733,11cyclictest7800-21cut20:15:162
16398994732,12cyclictest11684-21fschecks_time20:25:202
16398994731,13cyclictest19173-21grep20:45:252
1639799471,20cyclictest0-21swapper/122:17:101
211082460,1sleep30-21swapper/321:43:253
211082460,1sleep30-21swapper/321:43:253
16398994632,11cyclictest21856-21/usr/sbin/munin19:25:152
16397994622,10cyclictest7850irq/56-dwc_otg22:25:201
1639799461,4cyclictest0-21swapper/122:37:211
16397994610,25cyclictest0-21swapper/121:10:171
1640099458,4cyclictest7850irq/56-dwc_otg00:35:223
1640099457,25cyclictest0-21swapper/320:10:083
1640099447,25cyclictest0-21swapper/319:50:183
1640099447,22cyclictest0-21swapper/320:55:163
16400994437,5cyclictest37-21rcuc/300:25:153
72252420,2sleep10-21swapper/100:15:091
16400994218,4cyclictest4370-21grep21:20:363
1640099417,12cyclictest8050irq/56-dwc_otg_21:30:023
1640099414,22cyclictest0-21swapper/320:50:173
1640099409,16cyclictest0-21swapper/319:15:173
1640099407,23cyclictest0-21swapper/321:15:093
1640099407,23cyclictest0-21swapper/320:30:163
1640099407,23cyclictest0-21swapper/320:30:163
1640099402,33cyclictest0-21swapper/322:30:143
1640099402,24cyclictest0-21swapper/320:00:183
16397994017,10cyclictest18505-21ssh22:21:241
1640099394,8cyclictest11141-21latency_hist20:25:033
1640099391,6cyclictest18196-21ls00:30:243
16400993913,17cyclictest381ktimersoftd/319:26:343
1640099384,5cyclictest7850irq/56-dwc_otg22:10:323
1639799384,5cyclictest7950irq/56-dwc_otg_20:00:181
1640099377,15cyclictest8050irq/56-dwc_otg_22:25:253
1640099372,17cyclictest8050irq/56-dwc_otg_23:05:033
16397993713,5cyclictest26191-21if_err_wlan021:05:211
1640099369,4cyclictest0-21swapper/321:50:243
1640099364,8cyclictest19265-21ntpq20:45:263
1640099362,3cyclictest0-21swapper/321:05:203
1640099361,25cyclictest0-21swapper/323:55:173
1640099361,21cyclictest18543-21ls20:45:033
16400993610,5cyclictest29947-21date00:00:323
1639799368,15cyclictest0-21swapper/100:35:171
1639799364,6cyclictest23956-21tune2fs19:30:201
1639799361,5cyclictest668-21fschecks_count19:55:231
1640099351,6cyclictest32439-21cut22:40:233
1640099351,6cyclictest32439-21cut22:40:233
1640099351,2cyclictest0-21swapper/323:50:163
1640099351,21cyclictest623-21cpuspeed19:55:213
1639799355,6cyclictest8050irq/56-dwc_otg_19:25:301
1639799351,9cyclictest11463-21aten_rbpower_po20:25:171
74502341,4sleep37456-21ssh22:07:263
1640099347,14cyclictest0-21swapper/320:15:163
1640099345,14cyclictest8050irq/56-dwc_otg_23:15:203
1640099343,7cyclictest7950irq/56-dwc_otg_20:05:193
1640099343,5cyclictest7850irq/56-dwc_otg23:20:103
1640099342,5cyclictest11479-21ssh22:55:213
1640099341,6cyclictest2754-21ntp_states23:25:253
1640099341,5cyclictest7850irq/56-dwc_otg22:45:143
1640099341,4cyclictest8050irq/56-dwc_otg_19:20:133
1640099341,4cyclictest39-21ksoftirqd/323:30:233
16400993410,14cyclictest7950irq/56-dwc_otg_00:10:313
1639799347,7cyclictest144-21systemd-journal21:05:031
1639799344,7cyclictest19333-21sed19:15:321
16397993420,5cyclictest6393-21if_wlan020:10:221
16397993413,5cyclictest19334-21/usr/sbin/munin20:45:281
1640099339,5cyclictest7950irq/56-dwc_otg_21:35:333
1640099337,15cyclictest0-21swapper/319:35:093
1640099335,8cyclictest381ktimersoftd/300:20:163
1640099335,5cyclictest7850irq/56-dwc_otg21:45:313
1640099331,24cyclictest25408-21ls19:35:033
16400993312,4cyclictest0-21swapper/300:05:313
16400993311,5cyclictest0-21swapper/322:35:243
1640099324,5cyclictest0-21swapper/319:40:203
1640099324,5cyclictest0-21swapper/319:40:203
16400993213,5cyclictest661-21runrttasks20:25:303
1640099321,19cyclictest17404-21ssh22:20:253
1640099321,18cyclictest7850irq/56-dwc_otg23:05:313
1639799327,4cyclictest0-21swapper/123:45:471
1639799327,4cyclictest0-21swapper/123:45:471
1640099314,4cyclictest7950irq/56-dwc_otg_23:36:333
1640099312,3cyclictest7850irq/56-dwc_otg21:30:213
1640099311,16cyclictest32630-21kworker/u8:421:10:243
16400993110,6cyclictest8941-21sh00:16:313
1639799318,4cyclictest8050irq/56-dwc_otg_19:50:251
1639799315,5cyclictest16082-21sed20:35:321
1640099301,7cyclictest30744-21ntp_states21:55:303
16400993013,9cyclictest39-21ksoftirqd/322:50:353
1639799301,25cyclictest8787-21ls20:15:301
16400992910,6cyclictest39-21ksoftirqd/319:10:283
1639799291,5cyclictest27762-21fschecks_time19:40:241
1639799291,5cyclictest27762-21fschecks_time19:40:241
160912295,15sleep10-21swapper/119:09:361
1640099283,11cyclictest8050irq/56-dwc_otg_20:40:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional