You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 03:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Dec 14, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
479924212952,72sleep30-21swapper/319:09:293
47989928554,9sleep10-21swapper/119:09:111
47984028454,9sleep20-21swapper/219:08:242
47530328052,15sleep00-21swapper/019:05:090
480083992510,9cyclictest566502-21taskset20:30:163
480083992320,1cyclictest0-21swapper/323:20:153
48008399225,10cyclictest544791-21taskset20:10:013
8244782212,15sleep1824647-21/usr/sbin/munin00:35:151
48008399212,12cyclictest822823-21taskset00:30:193
480083992118,1cyclictest0-21swapper/321:50:253
480083992017,1cyclictest0-21swapper/320:55:133
48008399201,15cyclictest766198-21sleep23:40:013
48008399201,14cyclictest813374-21turbostat00:24:593
48008399201,11cyclictest559469-21taskset20:20:183
48008399192,12cyclictest624516-21taskset21:25:113
48008399192,11cyclictest771689-21taskset23:45:113
480083991916,1cyclictest0-21swapper/323:25:173
480083991916,1cyclictest0-21swapper/321:10:003
48008399191,14cyclictest492761-21chrt19:20:113
48008399188,6cyclictest719223-21taskset22:55:053
480083991816,1cyclictest0-21swapper/322:05:273
480083991816,1cyclictest0-21swapper/322:05:263
480083991815,1cyclictest0-21swapper/322:15:253
480083991815,1cyclictest0-21swapper/321:15:173
480083991814,2cyclictest0-21swapper/320:30:013
480083991814,2cyclictest0-21swapper/319:55:113
48008399181,13cyclictest734998-21chrt23:10:123
48008399181,13cyclictest734998-21chrt23:10:113
480083991811,2cyclictest0-21swapper/322:40:223
48008399181,11cyclictest649531-21taskset21:45:193
48008399181,11cyclictest482672-21taskset19:10:123
48008399180,11cyclictest587450-21taskset20:50:013
7472792171,11sleep0749091-21cut23:20:150
5989292171,2sleep00-21swapper/021:00:170
48008399178,6cyclictest744804-21taskset23:15:243
480083991715,1cyclictest0-21swapper/322:20:183
480083991715,1cyclictest0-21swapper/319:15:163
480083991714,1cyclictest0-21swapper/322:50:143
480083991714,1cyclictest0-21swapper/321:35:133
480083991714,1cyclictest0-21swapper/319:30:233
480083991714,1cyclictest0-21swapper/300:25:173
480083991713,2cyclictest0-21swapper/300:05:203
48008399171,12cyclictest697684-21sleep22:35:013
48008399171,12cyclictest676670-21turbostat22:15:013
48008399171,11cyclictest502771-21chrt19:28:403
48008399171,10cyclictest771380-21chrt23:44:383
48008399170,10cyclictest571860-21taskset20:35:143
6303502161,11sleep0631460-21grep21:30:140
5862682163,8sleep1586316-21ntpq20:45:221
5193682161,10sleep2519437-21kernelversion19:45:152
48008399169,4cyclictest5546222sleep320:15:273
48008399166,6cyclictest539465-21taskset20:04:273
48008399162,9cyclictest619715-21taskset21:20:153
48008399161,9cyclictest755969-21taskset23:30:073
48008399161,9cyclictest724496-21taskset23:00:053
48008399161,9cyclictest697896-21taskset22:35:003
48008399161,9cyclictest613431-21taskset21:11:013
48008399161,9cyclictest524127-21taskset19:50:013
480083991614,1cyclictest0-21swapper/322:45:153
480083991614,1cyclictest0-21swapper/300:10:163
480083991613,1cyclictest0-21swapper/321:31:073
480083991613,1cyclictest0-21swapper/320:10:123
480083991613,1cyclictest0-21swapper/319:45:153
480083991612,2cyclictest0-21swapper/321:40:253
480083991611,3cyclictest0-21swapper/320:50:113
48008399161,12cyclictest0-21swapper/300:35:173
48008399160,13cyclictest0-21swapper/319:40:193
48008399160,13cyclictest0-21swapper/319:35:113
48008399160,10cyclictest787968-21taskset00:00:133
48008399151,9cyclictest780966-21taskset23:50:193
48008399151,9cyclictest581670-21taskset20:40:463
48008399151,10cyclictest687548-21chrt22:25:103
48008399150,12cyclictest0-21swapper/323:05:123
48008399150,12cyclictest0-21swapper/322:00:143
48008399150,12cyclictest0-21swapper/321:00:203
7526472141,4sleep1754474-21grep23:25:181
6140302141,8sleep2614211-21cstates21:15:132
4802782141,8sleep2480505-21cstates19:10:092
48008399141,9cyclictest660904-21chrt21:59:143
480083991412,1cyclictest0-21swapper/300:15:213
480083991412,1cyclictest0-21swapper/300:15:203
48008399141,10cyclictest787096-21turbostat00:00:003
8126682131,8sleep1812823-21timerwakeupswit00:20:241
7949532131,8sleep1796518-21/usr/sbin/munin00:05:171
7667432131,8sleep2766953-21/usr/sbin/munin23:40:132
7405462131,8sleep1740197-21/usr/sbin/munin23:15:121
6984592130,8sleep0699030-21cut22:35:150
6049262131,8sleep1606724-21cut21:05:181
5967362131,4sleep0596787-21ldconfig20:55:190
5311442131,8sleep1532928-21cut19:55:171
8231492121,7sleep1823373-21unixbench_multi00:30:271
6917952121,7sleep0687413-21/usr/sbin/munin22:25:260
5763132121,4sleep0800-21snmpd20:38:230
5630662121,7sleep2564539-21/usr/sbin/munin20:25:162
5046092122,6sleep2506364-21kthreadcore19:30:172
7967082111,5sleep0796755-21ntpq00:05:200
7915232111,7sleep1791640-21/usr/sbin/munin00:00:211
7429352111,6sleep0744003-21memory23:15:150
6209552111,6sleep1621309-21kthreadcore21:20:181
5541672111,5sleep0554224-21ntpq20:15:190
5345812111,5sleep2534600-21apt-get20:00:102
5290892111,6sleep1529281-21latency_hist19:55:001
4852852111,6sleep1485333-21munin-run19:15:011
8031662101,4sleep1803185-21apt-get00:15:121
8031662101,4sleep1803185-21apt-get00:15:111
7721672101,4sleep2772238-21irqstats23:45:152
7512952100,6sleep0752645-21kthreadcore23:25:170
7389152101,4sleep2738961-21ntpq23:10:192
7389152101,4sleep2738961-21ntpq23:10:192
7183172101,5sleep2718585-21sed22:50:232
6670722101,5sleep0668695-21grep22:05:180
6670722101,5sleep0668695-21grep22:05:170
6650702101,5sleep2665183-21ntpq22:00:182
6136912100,1sleep10-21swapper/121:14:211
6087592101,4sleep0608778-21apt-get21:10:110
5124792101,5sleep0512603-21awk19:35:240
4877762101,4sleep0487799-21diskstats19:15:150
817664291,4sleep0813618-21/usr/sbin/munin00:25:200
801861291,3sleep1801899-21ntpq00:10:191
773505293,2sleep0221rcuc/023:45:160
766162291,4sleep1766346-21cat23:40:011
762260291,4sleep2763949-21cut23:35:172
725347293,3sleep0726994-21kthreadcore23:00:160
712682291,4sleep2712733-21ntpq22:45:192
687434290,5sleep244-21ksoftirqd/222:25:102
682764291,4sleep1684244-21tr22:20:171
675672291,4sleep1675728-21ntpq22:10:191
655990291,3sleep2656009-21apt-get21:55:102
645527290,3sleep2645546-21apt-get21:45:122
640572291,4sleep1640598-21fschecks_count21:40:131
629207290,1sleep00-21swapper/021:26:420
623620291,4sleep2623865-21users21:20:272
614008291,3sleep1614027-21apt-get21:15:121
603815291,4sleep0604925-21cut21:05:170
590966291,5sleep0468973-21kworker/0:0+events20:50:190
580963291,3sleep1581014-21ntpq20:40:181
576967291,3sleep2576450-21apt-get20:40:022
575489291,4sleep1575546-21ntpq20:35:201
570230290,1sleep10-21swapper/120:30:271
566877293,2sleep0568002-21kthreadcore20:30:190
559374291,3sleep2559462-21ntpq20:20:182
549973291,4sleep1550020-21systemctl20:15:001
549066291,4sleep2549112-21ls20:10:202
545551291,4sleep0545954-21cut20:10:140
538819291,4sleep0538941-21sed20:00:240
503708291,4sleep1504608-21cut19:30:151
496782290,6sleep0496862-21awk19:20:200
491485291,4sleep1491532-21ntpq19:15:241
827034281,3sleep0827925-21missed_timers00:35:180
806973281,3sleep0807112-21/usr/sbin/munin00:15:190
806973281,3sleep0807112-21/usr/sbin/munin00:15:180
803978281,3sleep2805619-21kthreadcore00:15:172
803978281,3sleep2805619-21kthreadcore00:15:162
798231281,3sleep0798394-21/usr/sbin/munin00:10:140
787897281,3sleep2787967-21irqstats00:00:142
787482281,3sleep0787397-21/usr/sbin/munin00:00:120
777553281,3sleep1779206-21kthreadcore23:50:151
760689280,1sleep10-21swapper/123:31:351
756542281,3sleep0757785-21kthreadcore23:30:130
731370281,3sleep0731788-21kthreadcore23:05:160
730202280,1sleep20-21swapper/223:05:152
730038282,3sleep1729750-21/usr/sbin/munin23:05:131
719804281,3sleep10-21swapper/122:55:161
717964280,3sleep0718019-21ntpq22:50:170
715461281,3sleep1717227-21grep22:50:141
708943281,3sleep0709156-21hddtemp_smartct22:45:130
708384281,3sleep1708127-21apt-get22:45:021
708116281,3sleep0151ktimers/022:45:000
697487280,1sleep20-21swapper/222:32:562
684245281,3sleep2685958-21latency22:20:192
676920281,3sleep1677085-21cstates22:15:131
674469280,1sleep20-21swapper/222:10:172
671665281,2sleep00-21swapper/022:10:100
665509281,3sleep0665514-21sensors_temp22:00:240
654713281,3sleep1654754-21ntpq21:50:191
649812281,3sleep0650002-21timerwakeupswit21:45:260
636439281,3sleep1638189-21kthreadcore21:35:171
635040280,3sleep0635058-21apt-get21:35:120
624494280,3sleep20-21swapper/221:25:112
623908280,1sleep00-21swapper/021:20:520
597067281,3sleep2597322-21awk20:55:252
592799280,1sleep10-21swapper/120:55:121
588091281,3sleep2589174-21kthreadcore20:50:172
571734280,1sleep20-21swapper/220:35:142
560725281,2sleep00-21swapper/020:25:090
556965282,2sleep0558712-21kthreadcore20:20:160
544054281,3sleep00-21swapper/020:05:260
508892281,3sleep1509044-21cut19:35:161
508397281,3sleep2508553-21cstates19:35:132
498522281,3sleep2499778-21cat19:25:152
497728280,1sleep00-21swapper/019:25:010
492739280,2sleep20-21swapper/219:20:112
824648270,3sleep2421rcuc/200:35:172
823829271,2sleep20-21swapper/200:35:002
820103271,3sleep0821818-21kthreadcore00:30:180
818351270,1sleep20-21swapper/200:26:482
809020271,2sleep0810537-21kthreadcore00:20:160
801900270,3sleep20-21swapper/200:10:202
782699271,3sleep2784243-21cut23:55:172
782095271,2sleep10-21swapper/123:55:111
767124270,3sleep0221rcuc/023:40:160
755666270,1sleep20-21swapper/223:29:282
750305270,4sleep10-21swapper/123:23:011
740196270,2sleep20-21swapper/223:15:092
729185270,1sleep20-21swapper/223:01:102
729179270,1sleep10-21swapper/123:01:051
724099270,1sleep00-21swapper/022:58:270
701957271,3sleep20-21swapper/222:35:212
701902271,2sleep10-21swapper/122:35:201
696733270,3sleep0696783-21/usr/sbin/munin22:30:220
696493271,3sleep1692674-21/usr/sbin/munin22:30:191
687412270,2sleep10-21swapper/122:25:101
649634271,3sleep10-21swapper/121:45:231
631461271,2sleep10-21swapper/121:30:151
592410270,1sleep10-21swapper/120:53:401
560403270,1sleep10-21swapper/120:24:141
518245270,1sleep10-21swapper/119:40:291
513627270,2sleep20-21swapper/219:40:112
502551270,1sleep10-21swapper/119:25:541
817231260,2sleep1817555-21strings00:25:181
813269260,1sleep20-21swapper/200:23:542
781749260,4sleep20-21swapper/223:53:562
776869260,1sleep00-21swapper/023:50:100
776300260,1sleep10-21swapper/123:46:061
770478261,2sleep10-21swapper/123:40:201
761208260,1sleep00-21swapper/023:35:030
760839260,1sleep20-21swapper/223:33:282
749693260,3sleep20-21swapper/223:20:212
739833260,1sleep10-21swapper/123:13:401
739833260,1sleep10-21swapper/123:13:391
739683260,1sleep00-21swapper/023:11:470
739683260,1sleep00-21swapper/023:11:460
707941260,4sleep10-21swapper/122:42:491
707136260,1sleep20-21swapper/222:40:212
681205261,2sleep20-21swapper/222:15:252
680869260,1sleep00-21swapper/022:15:200
671208260,1sleep20-21swapper/222:07:322
671208260,1sleep20-21swapper/222:07:312
671149260,1sleep10-21swapper/122:06:471
671149260,1sleep10-21swapper/122:06:461
664199261,2sleep10-21swapper/122:00:171
660378261,1sleep10-21swapper/121:55:251
651396261,2sleep20-21swapper/221:50:162
644923260,1sleep20-21swapper/221:40:382
635321261,2sleep20-21swapper/221:35:132
624790261,2sleep1624998-21hddtemp_smartct21:25:141
614304260,2sleep00-21swapper/021:15:150
602643260,4sleep20-21swapper/221:00:432
598310261,2sleep10-21swapper/121:00:141
587039260,1sleep00-21swapper/020:47:130
583868261,2sleep20-21swapper/220:45:192
581128261,2sleep00-21swapper/020:40:210
560747261,2sleep10-21swapper/120:25:111
550811260,2sleep1552242-21grep20:15:171
543701261,1sleep20-21swapper/220:05:202
538942261,3sleep10-21swapper/120:00:251
532929260,2sleep2533251-21strings19:55:182
529924261,2sleep00-21swapper/019:55:160
524748261,2sleep20-21swapper/219:50:162
523295260,1sleep10-21swapper/119:45:271
520942261,2sleep00-21swapper/019:45:180
514892261,2sleep00-21swapper/019:40:160
496535260,2sleep10-21swapper/119:20:181
491770261,2sleep20-21swapper/219:15:292
787070250,3sleep00-21swapper/023:59:510
719397250,1sleep20-21swapper/222:55:132
682182250,1sleep00-21swapper/022:20:110
655598250,3sleep00-21swapper/021:53:180
641219251,1sleep00-21swapper/021:40:160
634674250,1sleep20-21swapper/221:33:412
570453250,1sleep20-21swapper/220:30:282
540062250,1sleep10-21swapper/120:05:151
507926250,1sleep00-21swapper/019:32:380
484990250,3sleep00-21swapper/019:11:150
796519240,1sleep20-21swapper/200:05:182
656266240,1sleep00-21swapper/021:55:120
608750240,1sleep20-21swapper/221:10:052
603534240,1sleep20-21swapper/221:05:152
550021240,1sleep20-21swapper/220:15:022
524438240,1sleep00-21swapper/019:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional