You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Mon Nov 17, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2613175996353,7cyclictest2901849-21ntpq23:56:280
2613178996042,15cyclictest2923113-21apt-get00:21:181
26469612580,3chrt0-21swapper/019:41:280
26631102550,3chrt0-21swapper/119:56:261
2613178995543,9cyclictest2714159-21apt-get20:50:001
2613178995543,9cyclictest2714159-21apt-get20:50:001
2613178995537,15cyclictest2773091-21ntpq21:46:301
2613178995344,6cyclictest282-21systemd-journal23:51:491
261317599531,48cyclictest1-21systemd23:39:590
2613178995243,6cyclictest2917837-21latency_hist00:15:011
2613178994838,8cyclictest282-21systemd-journal23:40:001
2613175994636,7cyclictest2674416-21latency_hist20:10:010
2613178994538,5cyclictest2789272-21/usr/sbin/munin22:06:281
2613175994533,9cyclictest282-21systemd-journal22:00:000
2613175994437,5cyclictest1769938-21kworker/u8:3+flush-8:019:20:010
2613175994433,8cyclictest2625013-21kworker/u8:1+flush-8:020:01:190
2613178994328,12cyclictest282-21systemd-journal22:55:011
2613178994328,12cyclictest282-21systemd-journal00:16:181
2613178994328,12cyclictest282-21systemd-journal00:16:181
2613175994333,7cyclictest282-21systemd-journal21:55:000
2613178994231,9cyclictest282-21systemd-journal00:40:001
2613175994236,4cyclictest2674760-21apt-get20:11:150
2613178994120,18cyclictest2848270-21latency_hist23:05:011
2613175994131,7cyclictest2893467-21diskstats23:51:180
2613178994034,5cyclictest1769938-21kworker/u8:3+flush-8:022:41:491
2613178994026,12cyclictest2908308-21apt-get00:06:151
2613175994030,7cyclictest282-21systemd-journal20:21:490
2613175993935,3cyclictest1769938-21kworker/u8:3+flush-8:022:26:230
2613175993932,5cyclictest2625013-21kworker/u8:1+flush-8:020:16:330
2613178993829,7cyclictest2862194-21ntpq23:16:291
2613178993828,8cyclictest2901883-21ntpq23:56:281
2613178993828,8cyclictest2735009-21diskstats21:11:221
2613175993832,4cyclictest1769938-21kworker/u8:3+flush-8:023:31:280
2613175993831,5cyclictest2887902-21latency_hist23:45:010
2613175993829,6cyclictest2853496-21/usr/sbin/munin23:11:310
2613178993731,4cyclictest2625013-21kworker/u8:1+flush-8:019:26:331
2613175993732,3cyclictest1769938-21kworker/u8:3+flush-8:000:21:380
2613175993726,7cyclictest282-21systemd-journal23:00:010
261317599370,31cyclictest1206777-21kworker/0:1+events19:35:120
28160422360,2sleep3481ktimers/322:31:263
28160422360,2sleep3481ktimers/322:31:263
2613178993629,5cyclictest282-21systemd-journal20:31:501
2613178993624,10cyclictest2818566-21cron22:35:001
2613178993624,10cyclictest2818566-21cron22:35:001
2613175993627,6cyclictest2937567-21latency_hist00:35:010
260839323625,8sleep20-21swapper/219:05:112
2613178993524,9cyclictest2622537-21kthreadcore19:16:241
2613178993513,19cyclictest2753341-21ntpq21:26:311
261317599352,29cyclictest2818600-21kworker/0:2+events23:15:560
261317599351,30cyclictest2818600-21kworker/0:2+events22:40:040
261317599350,30cyclictest2818600-21kworker/0:2+events23:05:080
2613181993429,3cyclictest1769938-21kworker/u8:3+flush-8:019:11:362
2613181993425,7cyclictest282-21systemd-journal23:30:002
2613178993425,7cyclictest2645390-21diskstats19:41:221
2613178993416,15cyclictest282-21systemd-journal21:17:501
261317599343,30cyclictest2778963-21kworker/0:1+events22:03:280
2613175993428,4cyclictest2907970-21munin-plugin-st00:05:000
2613175993426,6cyclictest282-21systemd-journal20:43:090
2613175993424,8cyclictest2634033-21ntpq19:26:260
261317599341,30cyclictest1-21systemd19:35:010
261317899333,6cyclictest0-21swapper/120:16:151
2613178993328,3cyclictest2720219-21fschecks_time20:56:181
2613178993324,7cyclictest282-21systemd-journal21:44:501
261317899332,23cyclictest2650094-21apt-get19:46:151
2613175993327,4cyclictest2261296-21kworker/u8:0+flush-8:021:11:200
261317599331,31cyclictest2694423-21kworker/0:1+events20:49:200
261317599331,31cyclictest2694423-21kworker/0:1+events20:49:200
2613175993310,18cyclictest2788990-21kworker/0:0+events22:15:560
261317599330,27cyclictest2719469-21kworker/0:0+events21:35:440
261317899324,2cyclictest0-21swapper/122:26:151
2613178993222,8cyclictest2852274-21needreboot23:06:291
2613178993212,15cyclictest421rcuog/219:21:251
261317599328,19cyclictest2694423-21kworker/0:1+events20:35:040
2613175993227,3cyclictest2625013-21kworker/u8:1+events_unbound20:56:220
261317599321,30cyclictest2818600-21kworker/0:2+events23:27:440
261317599321,30cyclictest1206777-21kworker/0:1+events19:59:440
261317599321,30cyclictest1206777-21kworker/0:1+events19:23:120
261317599321,26cyclictest2818600-21kworker/0:2+events23:45:480
261317599321,26cyclictest2719469-21kworker/0:0+events21:15:520
261317599320,31cyclictest1206777-21kworker/0:1+events19:47:400
261317599320,26cyclictest2887918-21kworker/0:0+events00:10:240
27803292310,10sleep30-21swapper/321:56:273
2613181993123,6cyclictest282-21systemd-journal00:25:002
261317899313,5cyclictest0-21swapper/121:31:151
261317899313,5cyclictest0-21swapper/120:21:161
2613178993127,2cyclictest2642342-21kthreadcore19:36:261
2613178993126,3cyclictest2877109-21needreboot23:31:301
2613178993125,4cyclictest2903331-21apt-get00:01:151
2613178993124,5cyclictest2689247-21apt-get20:25:011
2613178993122,8cyclictest2635601-21cut19:31:211
2613178993117,12cyclictest2655317-21cut19:51:201
2613175993124,5cyclictest2823794-21kworker/u8:1+events_unbound22:46:310
261317599312,24cyclictest2719469-21kworker/0:0+events21:45:160
261317599311,29cyclictest2818600-21kworker/0:2+events22:53:040
261317599311,26cyclictest2694423-21kworker/0:1+events20:50:240
261317599311,26cyclictest2694423-21kworker/0:1+events20:50:240
261317599311,24cyclictest2719469-21kworker/0:0+events21:20:160
261317599310,30cyclictest2788990-21kworker/0:0+events22:39:040
261317599310,30cyclictest2719469-21kworker/0:0+events21:01:520
261317599310,26cyclictest2719469-21kworker/0:0+events21:05:040
261317899307,19cyclictest2764206-21cut21:40:011
261317899304,5cyclictest0-21swapper/121:51:151
261317899304,12cyclictest0-21swapper/121:57:521
2613178993021,7cyclictest2714739-21apt-get20:51:151
2613178993021,7cyclictest2714739-21apt-get20:51:151
2613178993016,12cyclictest829-21mta-sts-daemon23:21:351
2613178993011,17cyclictest0-21swapper/120:01:321
261317599303,26cyclictest2818600-21kworker/0:2+events23:24:000
261317599302,27cyclictest2694423-21kworker/0:1+events20:33:560
261317599302,23cyclictest2818600-21kworker/0:2+events23:00:320
261317599301,24cyclictest2887918-21kworker/0:0+events00:25:520
26129442302,6sleep00-21swapper/019:09:030
2613181992923,5cyclictest2261296-21kworker/u8:0+events_unbound22:51:322
2613181992923,4cyclictest2625013-21kworker/u8:1+flush-8:020:26:262
2613178992925,2cyclictest2803775-21cron22:20:001
2613178992924,3cyclictest2745371-21irqrtprio21:21:261
261317899292,11cyclictest2887064-21ntpq23:41:281
2613178992918,9cyclictest0-21swapper/120:06:291
2613178992917,10cyclictest0-21swapper/120:41:351
2613178992916,11cyclictest0-21swapper/122:11:351
2613178992916,11cyclictest0-21swapper/121:01:331
2613178992916,11cyclictest0-21swapper/100:26:191
2613178992912,14cyclictest2808136-21wc22:21:321
2613178992911,16cyclictest0-21swapper/120:11:341
2613175992923,3cyclictest2918259-21aten2_r5power_p00:16:180
2613175992923,3cyclictest2918259-21aten2_r5power_p00:16:180
261317599292,26cyclictest2788990-21kworker/0:0+events22:12:520
261317599292,22cyclictest2674437-21kworker/0:0+events20:25:040
261317599291,27cyclictest2719469-21kworker/0:0+events21:44:320
2613181992824,2cyclictest2719936-21aten2_r5power_c20:56:162
261317899283,4cyclictest0-21swapper/122:56:161
2613178992824,2cyclictest2788975-21systemd-run22:05:001
2613178992822,5cyclictest2853741-21cpuspeed_turbos23:11:221
2613178992817,9cyclictest2615634-21kthreadcore19:11:271
2613178992816,10cyclictest0-21swapper/120:40:001
2613178992815,11cyclictest0-21swapper/100:31:261
2613178992811,15cyclictest0-21swapper/123:26:331
261317599289,18cyclictest2749238-21kworker/0:2+events21:26:000
261317599283,24cyclictest2788990-21kworker/0:0+events22:31:400
261317599283,24cyclictest2788990-21kworker/0:0+events22:31:400
261317599283,20cyclictest2887918-21kworker/0:0+events00:05:080
2613175992823,3cyclictest2792578-21kthreadcore22:06:270
261317599282,25cyclictest2719469-21kworker/0:0+events21:32:560
261317599282,25cyclictest1206777-21kworker/0:1+events19:13:400
261317599281,26cyclictest2887918-21kworker/0:0+events00:38:160
2613181992722,4cyclictest2261296-21kworker/u8:0+flush-8:021:41:312
2613181992722,3cyclictest2679053-21unixbench_singl20:11:362
2613181992720,5cyclictest2942481-21journal-offline00:39:572
2613181992717,8cyclictest2719337-21apt-get20:55:002
2613181992717,8cyclictest2719337-21apt-get20:54:592
2613181992717,7cyclictest2902852-21cp00:00:002
261317899273,18cyclictest2818921-21apt-get22:36:151
261317899271,12cyclictest2828832-21apt-get22:46:161
2613178992710,15cyclictest0-21swapper/123:46:251
261317599272,24cyclictest1206777-21kworker/0:1+events19:50:440
2613181992622,3cyclictest2683906-21unixbench_multi20:16:352
2613181992621,4cyclictest1769938-21kworker/u8:3+flush-8:000:11:252
2613181992620,5cyclictest2625013-21kworker/u8:1+flush-8:021:01:212
261317899260,24cyclictest2733020-21cut21:06:261
2613181992522,2cyclictest2694813-21aten2_r5power_p20:31:172
2613181992521,2cyclictest2708910-21unixbench_singl20:41:372
2613181992520,4cyclictest2823794-21kworker/u8:1+flush-8:023:16:352
2613181992519,4cyclictest2261296-21kworker/u8:0+flush-8:020:36:262
2613181992518,5cyclictest2738784-21grep21:11:372
261317599252,22cyclictest2788990-21kworker/0:0+events22:21:560
2613181992421,2cyclictest2903416-21aten2_r5power_p00:01:172
2613181992421,2cyclictest2684914-21fschecks_count20:21:202
2613181992420,3cyclictest2798811-21idleruntime-cro22:15:002
2613181992419,3cyclictest2804169-21aten2_r5power_c22:21:152
2613181992419,3cyclictest2645492-21fschecks_time19:41:242
2613181992319,2cyclictest2660202-21fschecks_count19:56:202
2613181992319,2cyclictest2650179-21aten2_r5power_p19:46:182
2613181992315,6cyclictest2811221-21tr22:26:232
26087392237,10sleep10-21swapper/119:06:191
2613181992218,2cyclictest2784579-21cpuspeed_turbos22:01:202
2613181992216,4cyclictest2625006-21sendmail19:20:002
2613181992118,2cyclictest2759770-21fschecks_time21:36:222
2613181992117,2cyclictest2883545-21cpuspeed_turbos23:41:172
2613181992116,4cyclictest2751771-21kworker/u8:2+events_unbound21:26:272
2613181992115,4cyclictest2853049-21journal-offline23:08:352
261318199208,9cyclictest2793912-21awk22:10:002
2613181992017,2cyclictest2847948-21unixbench_singl23:01:372
2613181992016,2cyclictest2918232-21aten2_r5power_e00:16:182
2613181992016,2cyclictest2918232-21aten2_r5power_e00:16:172
2613181992015,3cyclictest2877690-21unixbench_singl23:31:392
2613181992015,3cyclictest0-21swapper/222:36:232
261318199201,2cyclictest26270001kthreadcore19:21:222
261318199201,14cyclictest2729819-21apt-get21:06:172
261318199192,12cyclictest2799160-21apt-get22:16:162
2613181991915,2cyclictest2756853-21kthreadcore21:31:252
2613181991814,3cyclictest2640449-21fschecks_count19:36:202
2613181991814,2cyclictest0-21swapper/221:55:002
2613181991814,2cyclictest0-21swapper/200:06:232
2613181991812,5cyclictest1769938-21kworker/u8:3+flush-8:021:21:312
2613181991812,4cyclictest0-21swapper/223:11:252
2613181991811,5cyclictest2890654-21runrttasks23:46:282
261318199181,13cyclictest2655003-21apt-get19:51:172
261318199181,12cyclictest2779410-21apt-get21:56:202
26088782183,8sleep30-21swapper/319:06:213
261318199175,4cyclictest0-21swapper/221:46:322
261318199172,10cyclictest2665814-21ps20:01:262
2613181991714,2cyclictest2893330-21aten2_r5power_v23:51:162
2613181991712,4cyclictest2872793-21kworker/u8:0+flush-8:023:36:282
261318199168,6cyclictest2814363-21uniq22:31:222
261318199168,6cyclictest2814363-21uniq22:31:222
261318199162,8cyclictest2932979-21apt-get00:31:202
261318199162,13cyclictest0-21swapper/223:21:122
261318199162,13cyclictest0-21swapper/222:42:512
261318199162,13cyclictest0-21swapper/221:16:022
2613181991613,2cyclictest0-21swapper/220:45:362
2613181991613,2cyclictest0-21swapper/220:45:362
2613181991613,2cyclictest0-21swapper/220:05:482
2613181991613,2cyclictest0-21swapper/219:27:122
261318199159,4cyclictest0-21swapper/200:26:242
261318199157,6cyclictest2840774-21tr22:56:312
2613181991511,2cyclictest0-21swapper/219:31:182
2613181991510,3cyclictest0-21swapper/222:46:262
27469972140,2sleep3471rcuc/321:21:273
26863002145,5sleep30-21swapper/320:21:253
28928922132,6sleep30-21swapper/323:50:013
28185642132,6sleep30-21swapper/322:35:013
27499382132,7sleep30-21swapper/321:26:223
27096912132,7sleep30-21swapper/320:46:173
27096912132,7sleep30-21swapper/320:46:173
26694672132,7sleep30-21swapper/320:05:003
29284472122,6sleep30-21swapper/300:26:213
29240092123,5sleep30-21swapper/300:21:273
29220052122,6sleep30-21swapper/300:16:333
29220052122,6sleep30-21swapper/300:16:333
29019222122,6sleep30-21swapper/323:56:313
28838442122,6sleep30-21swapper/323:41:223
28733782122,6sleep30-21swapper/323:31:193
28485892122,6sleep30-21swapper/323:06:193
28473912122,6sleep30-21swapper/323:01:293
28390812122,6sleep30-21swapper/322:56:273
28376572122,5sleep30-21swapper/322:51:343
28087512122,6sleep30-21swapper/322:25:013
27995352122,6sleep30-21swapper/322:16:233
27980932122,6sleep30-21swapper/322:11:323
27893012122,6sleep30-21swapper/322:06:193
27742022122,6sleep30-21swapper/321:50:503
27695332122,6sleep30-21swapper/321:46:213
27648712122,6sleep30-21swapper/321:41:233
27590962121,6sleep30-21swapper/321:35:013
27237092122,6sleep30-21swapper/320:56:293
27187442122,6sleep30-21swapper/320:51:353
27187442122,6sleep30-21swapper/320:51:353
27014682122,6sleep30-21swapper/320:36:263
26989612122,6sleep30-21swapper/320:31:373
26924772122,6sleep30-21swapper/320:26:273
26799572122,6sleep30-21swapper/320:16:203
26751962122,6sleep30-21swapper/320:11:223
26401892122,6sleep30-21swapper/319:36:183
26393052122,6sleep30-21swapper/319:31:373
26249662122,6sleep30-21swapper/319:20:013
26185122122,6sleep30-21swapper/319:15:043
26177752122,6sleep30-21swapper/319:11:393
29331422112,5sleep30-21swapper/300:31:213
29132492112,5sleep30-21swapper/300:11:203
29083122112,5sleep30-21swapper/300:06:183
29035862112,5sleep30-21swapper/300:01:213
28881682113,5sleep30-21swapper/323:45:393
28683312112,6sleep30-21swapper/323:25:333
28622042112,5sleep30-21swapper/323:16:303
28546162112,5sleep30-21swapper/323:11:273
28288362112,5sleep30-21swapper/322:46:163
28241162112,5sleep30-21swapper/322:41:193
28045292112,5sleep30-21swapper/322:21:203
27840362112,5sleep30-21swapper/322:00:013
27582302112,5sleep30-21swapper/321:31:283
27396982112,6sleep30-21swapper/321:16:193
27346672112,6sleep30-21swapper/321:10:253
27300632112,5sleep30-21swapper/321:06:203
27248902112,5sleep30-21swapper/321:01:183
27046142112,6sleep30-21swapper/320:41:163
26737532112,5sleep30-21swapper/320:06:373
26599082112,5sleep30-21swapper/319:56:133
26585952112,5sleep30-21swapper/319:51:283
26500982112,6sleep30-21swapper/319:46:183
26451452112,5sleep30-21swapper/319:41:203
29377902102,5sleep30-21swapper/300:35:033
28787242102,5sleep30-21swapper/323:36:203
28633132102,5sleep30-21swapper/323:20:033
26301982102,5sleep30-21swapper/319:25:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional