You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 12:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 30, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
828302212265,53sleep30-21swapper/319:05:383
92068821140,2sleep00-21swapper/020:30:270
82852828454,9sleep20-21swapper/219:08:302
82843328453,9sleep10-21swapper/119:07:221
82862028162,13sleep00-21swapper/019:09:410
828771993632,2cyclictest0-21swapper/319:59:593
828771992824,2cyclictest0-21swapper/323:30:233
828771992522,1cyclictest0-21swapper/323:40:163
828771992421,1cyclictest0-21swapper/320:09:593
12581302241,3sleep00-21swapper/000:18:510
82877199232,15cyclictest1207593-21taskset23:45:223
82877199222,15cyclictest1187644-21taskset23:35:113
828771992218,3cyclictest0-21swapper/322:55:013
828771992218,2cyclictest0-21swapper/300:35:143
828771992218,2cyclictest0-21swapper/300:35:143
828771992213,5cyclictest9743582sleep321:19:303
82877199221,16cyclictest1126881-21chrt22:55:193
82877199215,13cyclictest1005322-21taskset21:38:573
82877199212,14cyclictest1247844-21taskset00:10:233
82877199212,12cyclictest1097236-21taskset22:36:343
828771992117,2cyclictest0-21swapper/322:27:473
828771992111,6cyclictest0-21swapper/322:25:013
828771992110,7cyclictest1279341-21taskset00:32:163
82877199210,17cyclictest947277-21chrt20:55:443
82877199210,16cyclictest10202732sleep321:47:283
11228692211,12sleep2958484-21diskmemload22:53:462
82877199202,13cyclictest1090060-21taskset22:32:423
82877199202,12cyclictest1217346-21taskset23:52:073
82877199202,12cyclictest1060667-21taskset22:15:093
828771992017,1cyclictest0-21swapper/323:00:263
828771992016,2cyclictest0-21swapper/322:10:173
828771992016,2cyclictest0-21swapper/300:20:233
828771992015,2cyclictest0-21swapper/323:25:453
828771992012,5cyclictest66550irq/131-enp1s0-TxRx-321:50:293
828771992012,5cyclictest66550irq/131-enp1s0-TxRx-321:25:283
82877199201,15cyclictest1106268-21chrt22:43:163
12101232201,4sleep1958484-21diskmemload23:48:251
82877199197,8cyclictest990880-21chrt21:30:123
82877199192,13cyclictest1139916-21chrt23:05:133
82877199192,12cyclictest1234494-21taskset00:03:223
82877199192,12cyclictest1169767-21taskset23:22:253
82877199192,12cyclictest1012395-21taskset21:42:553
828771991914,2cyclictest0-21swapper/300:15:173
828771991913,3cyclictest888990-21taskset20:00:413
82877199191,14cyclictest981249-21chrt21:22:143
82877199191,14cyclictest0-21swapper/322:45:253
82877199191,14cyclictest0-21swapper/322:45:243
82877199191,13cyclictest959144-21taskset21:10:153
82877199191,11cyclictest854585-21taskset19:30:193
82877199191,11cyclictest1151404-21taskset23:10:183
11933852191,12sleep01193397-21ssh23:37:020
10037462191,9sleep21003758-21rm21:36:042
9950342181,11sleep0995094-21ntpq21:30:220
9896182181,8sleep1989628-21ssh21:28:251
9727362181,4sleep0958484-21diskmemload21:16:510
8883282181,12sleep0888375-21ntpq20:00:220
82877199187,8cyclictest1225217-21taskset23:56:483
82877199182,11cyclictest1037483-21taskset22:00:153
828771991816,1cyclictest0-21swapper/323:15:243
828771991814,2cyclictest0-21swapper/320:10:283
828771991812,3cyclictest0-21swapper/300:26:203
82877199181,14cyclictest0-21swapper/321:55:573
82877199181,14cyclictest0-21swapper/320:35:133
82877199181,13cyclictest0-21swapper/320:20:233
82877199180,15cyclictest0-21swapper/300:05:083
82877199180,11cyclictest873240-21taskset19:46:463
12276852181,4sleep0958484-21diskmemload00:00:100
12175172181,3sleep10-21swapper/123:52:261
12155752181,3sleep00-21swapper/023:50:200
11857462181,5sleep21185757-21ssh23:32:312
11228962181,2sleep00-21swapper/022:53:480
9975792171,4sleep2958484-21diskmemload21:33:492
82877199179,4cyclictest10501152sleep322:05:363
828771991715,1cyclictest0-21swapper/319:15:183
828771991714,2cyclictest0-21swapper/319:25:263
828771991714,1cyclictest0-21swapper/321:05:193
828771991714,1cyclictest0-21swapper/321:05:193
828771991714,1cyclictest0-21swapper/319:10:273
828771991713,2cyclictest0-21swapper/319:25:013
82877199170,13cyclictest0-21swapper/321:00:233
82877199170,13cyclictest0-21swapper/320:45:283
12412932171,3sleep10-21swapper/100:06:421
12117632171,10sleep21211478-21/usr/sbin/munin23:50:152
11936112171,4sleep1958484-21diskmemload23:37:271
11881442178,5sleep21188239-21uname23:35:162
11624952171,3sleep00-21swapper/023:18:300
11525002171,2sleep00-21swapper/023:10:320
11371152171,4sleep1958484-21diskmemload23:01:071
10733602171,10sleep1958484-21diskmemload22:20:421
10580752171,2sleep00-21swapper/022:10:580
10362452171,3sleep20-21swapper/221:58:372
82877199162,9cyclictest921126-21taskset20:35:003
82877199161,9cyclictest873747-21taskset19:50:033
82877199161,9cyclictest861796-21taskset19:35:193
828771991613,1cyclictest0-21swapper/320:15:173
12788192161,2sleep10-21swapper/100:31:101
12507772160,9sleep10-21swapper/100:14:301
12413002160,3sleep2958484-21diskmemload00:06:432
11817732166,2sleep0171rcu_preempt23:30:160
11455962161,10sleep01145605-21ssh23:07:150
11308852168,3sleep1958484-21diskmemload22:58:531
10893032160,4sleep00-21swapper/022:31:180
10819752160,3sleep00-21swapper/022:27:140
10742192161,3sleep2958484-21diskmemload22:22:112
9376142151,9sleep2937326-21/usr/sbin/munin20:50:152
828771991512,1cyclictest0-21swapper/320:25:153
82877199151,10cyclictest863162-21chrt19:40:133
82877199150,13cyclictest0-21swapper/320:40:193
12651762151,8sleep0958484-21diskmemload00:22:310
12332122158,1sleep10-21swapper/100:01:231
12092602151,2sleep00-21swapper/023:47:040
11780602150,2sleep10-21swapper/123:27:531
11627402151,9sleep1958484-21diskmemload23:18:541
11397512151,4sleep2958484-21diskmemload23:05:122
11239822151,3sleep01123983-21ssh22:55:120
11008732155,6sleep01102482-21cut22:40:160
10981022151,2sleep10-21swapper/122:37:581
10199622150,2sleep20-21swapper/221:46:552
10142962151,7sleep11014556-21df21:45:141
10135502151,2sleep10-21swapper/121:44:501
8289642141,8sleep0829130-21expr19:10:120
82877199147,2cyclictest0-21swapper/320:50:123
12672322141,8sleep01267385-21gltestperf00:25:170
12017402141,6sleep00-21swapper/023:42:380
12008352141,9sleep21200837-21ssh23:41:092
10991842141,8sleep0958484-21diskmemload22:39:370
10804072141,2sleep10-21swapper/122:25:231
10535142141,8sleep11052980-21/usr/sbin/munin22:10:161
9950962131,7sleep1995193-21ps21:30:231
9431952135,4sleep0944652-21cut20:55:180
9378972131,7sleep0939173-21cat20:50:160
8857642131,8sleep1887495-21cut20:00:201
12497342131,3sleep21249744-21ssh00:12:482
12246332131,5sleep0958484-21diskmemload23:55:540
11443652131,6sleep11144556-21/usr/sbin/munin23:05:271
11071032131,5sleep10-21swapper/122:44:431
10891232131,5sleep10-21swapper/122:31:001
10663532131,6sleep10-21swapper/122:16:571
10433372131,3sleep1958484-21diskmemload22:02:221
9882652121,3sleep2988280-21ssh21:26:022
9585182121,4sleep0151ktimers/021:10:010
8948792120,7sleep2895034-21cstates20:10:152
8531092121,7sleep2854583-21kthreadcore19:30:192
12861042121,4sleep1958484-21diskmemload00:36:201
12861042121,4sleep1958484-21diskmemload00:36:191
12519922121,3sleep21252606-21kthreadcore00:15:162
12335262120,8sleep21233536-21ssh00:01:532
12259762121,7sleep21225989-21ssh23:58:012
12075502121,5sleep21207591-21ntpq23:45:222
11526902120,2sleep20-21swapper/223:10:542
11524672121,7sleep1958484-21diskmemload23:10:291
11252442121,6sleep21126878-21kthreadcore22:55:182
11139102120,1sleep20-21swapper/222:47:452
11139102120,1sleep20-21swapper/222:47:452
10908352120,1sleep2958484-21diskmemload22:34:062
10363972121,5sleep00-21swapper/021:58:520
10286092121,4sleep1958484-21diskmemload21:53:401
9830732111,4sleep0958484-21diskmemload21:25:090
9810812111,4sleep0958484-21diskmemload21:22:000
9805872111,3sleep1958484-21diskmemload21:21:091
9753262112,4sleep2975351-21apt-get21:20:142
9740662111,4sleep2958484-21diskmemload21:19:002
9721752111,3sleep10-21swapper/121:15:551
9651012111,3sleep20-21swapper/221:12:482
9638472111,4sleep1958484-21diskmemload21:10:421
8988432111,5sleep1898894-21ntpq20:10:221
8895952112,4sleep2889613-21apt-get20:05:142
8684522111,4sleep2868470-21apt-get19:45:132
8456852111,5sleep2845736-21ntpq19:20:212
12872112111,3sleep00-21swapper/000:38:560
12872112111,3sleep00-21swapper/000:38:560
12862782111,3sleep20-21swapper/200:36:412
12862782111,3sleep20-21swapper/200:36:402
12797292111,6sleep2958484-21diskmemload00:33:082
12786032111,3sleep0958484-21diskmemload00:30:420
12727712111,4sleep1958484-21diskmemload00:28:181
12717932111,4sleep2958484-21diskmemload00:25:392
12645322111,3sleep2958484-21diskmemload00:21:262
12569342111,2sleep10-21swapper/100:16:441
12492342111,3sleep00-21swapper/000:11:570
12245962110,3sleep11224607-21ssh23:55:511
11790402111,4sleep0958484-21diskmemload23:29:320
11770162111,5sleep20-21swapper/223:26:092
11622112111,4sleep2958484-21diskmemload23:18:032
11382712111,4sleep2958484-21diskmemload23:03:082
11319532111,3sleep00-21swapper/023:00:140
11131952111,4sleep0958484-21diskmemload22:46:350
11131952111,4sleep0958484-21diskmemload22:46:340
11052972111,4sleep2958484-21diskmemload22:41:362
10972272111,3sleep20-21swapper/222:36:352
10826312111,3sleep20-21swapper/222:28:212
10648822111,4sleep01064936-21ntpq22:15:240
10606662111,5sleep2958484-21diskmemload22:15:092
10594792111,3sleep20-21swapper/222:13:292
10508512111,3sleep20-21swapper/222:06:562
10442932111,4sleep0958484-21diskmemload22:04:050
10428252111,4sleep2958484-21diskmemload22:01:312
10282182111,4sleep2958484-21diskmemload21:52:572
10142622112,4sleep01014284-21apt-get21:45:130
10068922111,5sleep21007638-21grep21:40:182
10045352111,5sleep0958484-21diskmemload21:37:350
9537442101,4sleep1953954-21hddtemp_smartct21:05:171
9537442101,4sleep1953954-21hddtemp_smartct21:05:161
9534512101,3sleep2953470-21apt-get21:05:142
9534512101,3sleep2953470-21apt-get21:05:132
9427792101,4sleep1942902-21diskstats20:55:151
9125892101,4sleep2914366-21tr20:25:182
9060002101,4sleep2906071-21irqstats20:20:172
8633062101,4sleep0863429-21diskstats19:40:140
12642462101,2sleep10-21swapper/100:20:561
12011692101,2sleep10-21swapper/123:41:441
11852512101,3sleep1958484-21diskmemload23:31:421
11697562101,3sleep20-21swapper/223:22:252
11686522101,3sleep10-21swapper/123:20:351
11637362101,3sleep0958484-21diskmemload23:20:100
11218812101,3sleep10-21swapper/122:52:131
11127832101,3sleep10-21swapper/122:45:561
11127832101,3sleep10-21swapper/122:45:551
10735182101,3sleep0958484-21diskmemload22:20:570
10512982101,3sleep00-21swapper/022:07:420
10507862101,4sleep1958484-21diskmemload22:06:511
10296802101,3sleep11029709-21apt-get21:55:121
10282832101,3sleep0958484-21diskmemload21:53:030
10036052100,3sleep1958484-21diskmemload21:35:521
952911291,4sleep01-21systemd21:04:590
948191291,4sleep1948400-21hddtemp_smartct21:00:151
947901291,3sleep2947920-21apt-get21:00:122
928893291,5sleep1930577-21latency20:40:191
909131290,1sleep10-21swapper/120:20:191
884614291,4sleep2884740-21perl20:00:172
882635291,4sleep1882955-21grep19:55:191
878128291,4sleep2878398-21head19:50:272
862165291,4sleep1862234-21threads19:35:271
861371291,4sleep2861793-21grep19:35:192
850921291,4sleep0850971-21ldconfig19:25:200
840668291,4sleep2840829-21sed19:15:272
836311291,4sleep1836497-21expr19:15:151
836289291,3sleep0836308-21apt-get19:15:140
1241402291,4sleep01241412-21ssh00:06:530
1009267291,4sleep01010225-21memory21:40:200
935970281,2sleep10-21swapper/120:45:211
921386281,2sleep20-21swapper/220:35:122
920115280,1sleep10-21swapper/120:30:211
901831281,3sleep0902233-21kthreadcore20:15:180
900432281,3sleep1900605-21gltestperf20:15:151
894553280,1sleep00-21swapper/020:09:290
883355281,3sleep0883421-21sed19:55:260
939175271,3sleep1940953-21kthreadcore20:50:171
936913270,1sleep20-21swapper/220:48:072
932307271,3sleep00-21swapper/020:45:160
926394271,3sleep0802-21snmpd20:39:390
914814271,2sleep10-21swapper/120:25:211
910765271,2sleep0910784-21apt-get20:25:120
899616270,4sleep00-21swapper/020:11:410
875437271,2sleep10-21swapper/119:50:181
872366271,2sleep1872483-21ntpq19:45:211
866964271,3sleep10-21swapper/119:40:201
852506270,2sleep10-21swapper/119:30:141
852505270,1sleep00-21swapper/019:30:120
845791271,3sleep00-21swapper/019:20:220
841642270,2sleep10-21swapper/119:20:121
833849270,1sleep20-21swapper/219:12:442
955524261,2sleep00-21swapper/021:05:200
955524261,2sleep00-21swapper/021:05:200
942612260,1sleep20-21swapper/220:55:122
927313261,2sleep00-21swapper/020:40:180
925199260,1sleep1925372-21strings20:35:191
920818260,1sleep20-21swapper/220:31:122
907344260,2sleep0221rcuc/020:20:180
874327261,2sleep00-21swapper/019:50:160
873149260,1sleep00-21swapper/019:45:360
858380261,2sleep00-21swapper/019:35:180
851963260,3sleep2828764-21cyclictest19:29:592
847515260,2sleep1849007-21ps19:25:171
931201250,1sleep20-21swapper/220:40:292
900606250,1sleep20-21swapper/220:15:162
879543250,1sleep20-21swapper/219:55:162
863140250,1sleep20-21swapper/219:40:132
889381240,1sleep10-21swapper/120:05:011
831035240,1sleep10-21swapper/119:10:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional