You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 02:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Nov 21, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1399262213254,71sleep30-21swapper/319:05:173
140101328960,6sleep00-21swapper/019:08:560
140080828655,10sleep20-21swapper/219:06:212
140081227864,9sleep10-21swapper/119:06:251
17163512340,1sleep00-21swapper/023:05:200
17163512340,1sleep00-21swapper/023:05:200
1401203993420,10cyclictest1720935-21sleep23:10:003
1401203993420,10cyclictest1720935-21sleep23:10:003
1401203993222,7cyclictest1855738-21turbostat00:35:013
140120399320,17cyclictest16168612sleep322:02:553
1401203993127,2cyclictest0-21swapper/323:30:013
16648482301,3sleep00-21swapper/022:34:510
1401203992925,2cyclictest0-21swapper/323:30:163
1401203992820,4cyclictest66550irq/131-enp1s0-TxRx-321:44:263
1401203992814,10cyclictest1632337-21chrt22:12:363
1401203992723,2cyclictest0-21swapper/323:40:253
1401203992723,2cyclictest0-21swapper/321:22:263
17828992261,20sleep11530878-21diskmemload23:46:401
17574732251,4sleep20-21swapper/223:30:202
1401203992522,1cyclictest0-21swapper/322:49:043
1401203992522,1cyclictest0-21swapper/320:35:013
1401203992522,1cyclictest0-21swapper/300:25:133
1401203992513,9cyclictest0-21swapper/323:13:283
1401203992512,2cyclictest541ktimers/321:35:113
1401203992421,1cyclictest0-21swapper/321:00:003
1401203992420,2cyclictest0-21swapper/323:19:013
1401203992420,1cyclictest0-21swapper/321:10:243
1401203992420,1cyclictest0-21swapper/321:10:243
1401203992413,8cyclictest0-21swapper/300:36:323
1401203992412,9cyclictest0-21swapper/322:27:573
1401203992412,2cyclictest541ktimers/322:43:483
1401203992412,2cyclictest541ktimers/321:26:343
140120399238,6cyclictest541ktimers/322:23:283
140120399236,13cyclictest1790707-21chrt23:51:063
1401203992321,1cyclictest0-21swapper/322:05:153
1401203992320,1cyclictest0-21swapper/322:19:113
1401203992320,1cyclictest0-21swapper/321:45:013
1401203992314,7cyclictest0-21swapper/323:20:173
1401203992311,2cyclictest541ktimers/321:17:333
1401203992311,2cyclictest541ktimers/300:16:473
1401203992311,2cyclictest541ktimers/300:01:273
1401203992311,2cyclictest541ktimers/300:01:263
1401203992310,2cyclictest0-21swapper/321:33:413
1401203992310,2cyclictest0-21swapper/300:11:053
15877472228,9sleep21589220-21cut21:45:182
15634002222,8sleep11563425-21apt-get21:30:141
1401203992219,1cyclictest0-21swapper/320:25:173
1401203992219,1cyclictest0-21swapper/319:20:003
1401203992218,2cyclictest0-21swapper/323:37:313
1401203992218,2cyclictest0-21swapper/322:52:163
1401203992218,2cyclictest0-21swapper/322:39:413
1401203992218,2cyclictest0-21swapper/321:55:263
1401203992218,2cyclictest0-21swapper/300:21:533
1401203992212,8cyclictest0-21swapper/322:55:203
140120399221,18cyclictest1814995-21taskset00:06:133
17685252211,8sleep01530878-21diskmemload23:39:370
17503102211,5sleep21530878-21diskmemload23:25:362
1401203992118,2cyclictest0-21swapper/320:00:143
1401203992118,1cyclictest0-21swapper/320:50:273
1401203992118,1cyclictest0-21swapper/320:45:213
1401203992112,6cyclictest0-21swapper/321:51:113
140120399211,17cyclictest0-21swapper/322:33:143
140120399211,15cyclictest1530885-21turbostat21:10:003
1401203992110,2cyclictest541ktimers/323:55:173
18085532201,7sleep11530878-21diskmemload00:03:351
18085532201,7sleep11530878-21diskmemload00:03:351
140120399208,3cyclictest541ktimers/323:45:043
140120399202,12cyclictest1710374-21taskset23:00:463
1401203992018,1cyclictest0-21swapper/320:40:183
17122812191,11sleep11530878-21diskmemload23:04:021
16062482191,9sleep21606761-21cut21:55:202
140120399194,13cyclictest0-21swapper/320:19:553
1401203991917,1cyclictest0-21swapper/320:25:013
1401203991917,1cyclictest0-21swapper/319:45:153
1401203991915,2cyclictest0-21swapper/320:15:003
17590262181,3sleep10-21swapper/123:31:591
16879492181,4sleep21530878-21diskmemload22:48:112
16720842181,2sleep00-21swapper/022:38:440
16629342180,3sleep11662943-21ssh22:31:311
15775322181,5sleep21530878-21diskmemload21:37:482
1401203991815,2cyclictest0-21swapper/319:55:233
1401203991815,1cyclictest0-21swapper/319:35:133
1401203991814,2cyclictest0-21swapper/319:27:273
18160412171,8sleep01530878-21diskmemload00:08:110
17919182171,4sleep11530878-21diskmemload23:52:571
16956732171,4sleep11530878-21diskmemload22:52:521
16637332171,4sleep21530878-21diskmemload22:32:522
16482792171,4sleep11530878-21diskmemload22:23:091
16397212171,4sleep01530878-21diskmemload22:16:500
16328732171,10sleep21530878-21diskmemload22:13:282
15449132175,4sleep2171rcu_preempt21:16:322
15448102171,3sleep10-21swapper/121:16:221
1401203991714,2cyclictest0-21swapper/320:05:193
1401203991714,1cyclictest0-21swapper/319:20:263
1401203991713,2cyclictest0-21swapper/319:40:093
140120399171,10cyclictest1404995-21taskset19:10:193
18384422161,3sleep10-21swapper/100:22:031
18373462161,8sleep21837595-21users00:20:282
18246662161,8sleep10-21swapper/100:14:461
17993152161,3sleep11530878-21diskmemload23:56:381
17344802161,3sleep10-21swapper/123:15:571
16894612161,9sleep01689721-21df_inode22:50:140
140120399163,1cyclictest0-21swapper/320:35:183
1401203991614,1cyclictest0-21swapper/321:04:173
1401203991613,2cyclictest0-21swapper/319:30:203
1401203991613,1cyclictest0-21swapper/319:50:163
18604732151,9sleep01860523-21ntpq00:35:210
18473852151,7sleep21530878-21diskmemload00:29:062
18334482151,3sleep01834874-21kthreadcore00:20:180
18294412151,8sleep01829535-21ps00:15:250
18293182151,4sleep21829367-21ntpq00:15:222
18246742151,9sleep21530878-21diskmemload00:14:472
18246212151,9sleep01824633-21ssh00:14:410
18162462151,7sleep244-21ksoftirqd/200:08:322
17694852151,6sleep21769700-21hddtemp_smartct23:40:142
17377292151,4sleep11737949-21hddtemp_smartct23:20:171
17320072151,9sleep21733317-21load23:15:182
16791012152,9sleep00-21swapper/022:42:080
16783582151,9sleep21530878-21diskmemload22:40:542
16727742151,9sleep21672785-21sh22:39:532
16409172151,2sleep20-21swapper/222:18:572
16311752151,3sleep01530878-21diskmemload22:10:370
16111482151,9sleep11611736-21cut22:00:161
15697032151,7sleep21530878-21diskmemload21:32:592
15608132151,9sleep21530878-21diskmemload21:26:102
15547902151,9sleep11554799-21ssh21:24:481
15468772151,10sleep01546888-21ssh21:19:410
14626972151,9sleep11464095-21cut20:05:161
14019652151,9sleep11403180-21cut19:10:181
18555652141,3sleep21855577-21ssh00:34:472
18472142141,7sleep11530878-21diskmemload00:28:481
18316082141,7sleep11530878-21diskmemload00:18:301
17841512141,6sleep20-21swapper/223:48:332
17450692140,7sleep01530878-21diskmemload23:25:020
17132742141,8sleep11713295-21apt-get23:05:141
17132742141,8sleep11713295-21apt-get23:05:131
17118662141,6sleep20-21swapper/223:03:192
16190512141,8sleep01620222-21cut22:05:150
15874182141,9sleep01587637-21smartctl21:45:150
15540612141,9sleep01530878-21diskmemload21:23:350
15376792141,8sleep11537689-21ssh21:12:581
15376792141,8sleep11537689-21ssh21:12:571
14984292141,9sleep01498434-21/usr/sbin/munin20:35:250
14671352141,9sleep01467182-21systemd-run20:10:000
14618332141,9sleep2802-21snmpd20:04:372
17977572131,6sleep01530878-21diskmemload23:55:210
17761952131,7sleep11530878-21diskmemload23:43:591
17585222131,8sleep01758536-21ssh23:31:070
17515782131,2sleep10-21swapper/123:27:431
16007602131,5sleep10-21swapper/121:52:181
15798002130,8sleep21581073-21cut21:40:172
15711652131,8sleep11530878-21diskmemload21:35:021
14386862131,8sleep11435475-21/usr/sbin/munin19:40:151
14249322132,5sleep01424951-21apt-get19:30:110
17835112120,8sleep01783522-21ssh23:47:350
17733412122,5sleep01773393-21ntpq23:40:210
17666062121,5sleep10-21swapper/123:36:211
17270102121,5sleep00-21swapper/023:11:510
17263962121,3sleep10-21swapper/123:10:511
16805042120,5sleep10-21swapper/122:44:261
16497502121,6sleep11649782-21ssh22:25:131
16312362121,5sleep11530878-21diskmemload22:10:421
16000502121,5sleep21530878-21diskmemload21:51:072
15870932121,5sleep11587118-21apt-get21:45:131
15843042121,5sleep01530878-21diskmemload21:40:550
15318282121,7sleep01532887-21kthreadcore21:10:160
15318282121,7sleep01532887-21kthreadcore21:10:150
15258082120,2sleep10-21swapper/121:05:121
15099232121,7sleep11510045-21tr20:50:131
14722442120,7sleep072-21kcompactd020:12:570
14249542121,6sleep11425173-21cstates19:30:131
14098382121,7sleep01411538-21kthreadcore19:15:170
18625872111,3sleep20-21swapper/200:37:592
18546742111,4sleep01530878-21diskmemload00:33:170
18534892111,4sleep11530878-21diskmemload00:31:051
18465952111,4sleep01530878-21diskmemload00:27:480
18163572111,3sleep11530878-21diskmemload00:08:441
18077552111,3sleep20-21swapper/200:02:162
18077552111,3sleep20-21swapper/200:02:152
18068602111,3sleep00-21swapper/000:00:460
18068602111,3sleep00-21swapper/000:00:460
17911952110,3sleep00-21swapper/023:51:520
17425382111,3sleep20-21swapper/223:20:552
17416522111,5sleep01741702-21ssh23:20:250
17356812111,3sleep00-21swapper/023:17:520
17281732111,3sleep21530878-21diskmemload23:13:412
17190952111,3sleep20-21swapper/223:07:082
17190952111,3sleep20-21swapper/223:07:082
17094472111,5sleep01709501-21ntpq23:00:200
17039372111,3sleep00-21swapper/022:58:200
16973742112,4sleep21697402-21apt-get22:55:132
16957222111,4sleep21530878-21diskmemload22:52:572
16872732111,4sleep11530878-21diskmemload22:47:011
16653322111,3sleep10-21swapper/122:35:121
16459042111,4sleep21530878-21diskmemload22:20:222
16408592111,3sleep10-21swapper/122:18:471
16253862111,4sleep11530878-21diskmemload22:09:051
16168342111,6sleep01530878-21diskmemload22:02:520
16084582111,4sleep10-21swapper/121:57:031
15847662111,3sleep10-21swapper/121:41:441
15706002110,2sleep00-21swapper/021:34:300
15608862110,1sleep00-21swapper/021:26:150
15527012111,6sleep21552706-21ssh21:21:132
15364262111,3sleep20-21swapper/221:11:022
15364262111,3sleep20-21swapper/221:11:022
14555862111,4sleep01455632-21ntpq19:55:210
14457162110,2sleep00-21swapper/019:46:040
14301882112,4sleep01430207-21apt-get19:35:130
17999972101,3sleep21530878-21diskmemload23:57:432
17910412100,2sleep20-21swapper/223:51:372
17670372101,3sleep20-21swapper/223:37:062
17023242101,3sleep11530878-21diskmemload22:55:351
16868642101,3sleep00-21swapper/022:46:210
16569772101,2sleep20-21swapper/222:29:432
16558482100,3sleep00-21swapper/022:27:410
16428882101,5sleep01644520-21cut22:20:200
16241692100,2sleep20-21swapper/222:06:592
16105992101,4sleep21610864-21df22:00:132
16080342101,2sleep00-21swapper/021:56:210
15998282101,3sleep00-21swapper/021:50:410
15782822101,3sleep00-21swapper/021:39:060
15616162100,1sleep10-21swapper/121:27:321
15151802101,3sleep10-21swapper/120:55:181
15043702101,5sleep21504573-21sed20:45:012
14924582101,4sleep11492515-21ntpq20:30:251
14818892101,4sleep21481940-21ntpq20:20:202
14595572101,4sleep11460707-21memory20:00:191
14517272101,4sleep11451850-21diskstats19:55:141
14503272101,4sleep01450382-21ld19:50:200
14438072101,4sleep21444870-21memory19:45:162
14395132101,4sleep01439567-21ntpq19:40:180
14197022101,4sleep21419920-21df_inode19:25:132
14140352101,4sleep21414061-21munin-run19:20:002
1856286290,2sleep11530878-21diskmemload00:35:061
1520205291,3sleep21520277-21munin-run21:00:002
1509640291,4sleep11509759-21sed20:50:011
1509298290,1sleep20-21swapper/220:45:412
1440469291,4sleep21440930-21cut19:45:012
1429231291,4sleep21429437-21grep19:30:262
1419376291,5sleep11419377-21cron19:25:001
1414317291,3sleep01414336-21apt-get19:20:090
1401893291,4sleep01401964-21/usr/sbin/munin19:10:170
1497964281,3sleep21498105-21ntpq20:35:192
1489088281,3sleep21490502-21tr20:30:222
1477657283,3sleep2246-21systemd-journal20:20:002
1442091281,3sleep10-21swapper/119:45:161
1414482280,1sleep20-21swapper/219:20:122
1407156281,3sleep11406441-21apt-get19:15:021
1530729270,1sleep00-21swapper/021:08:200
1525518270,1sleep00-21swapper/021:04:380
1525300270,1sleep10-21swapper/121:01:551
1520490270,2sleep20-21swapper/221:00:122
1514623270,1sleep00-21swapper/020:51:230
1504214270,1sleep10-21swapper/120:43:041
1504096270,1sleep00-21swapper/020:41:340
1498435271,3sleep10-21swapper/120:35:261
1487894270,1sleep00-21swapper/020:25:310
1483193270,2sleep20-21swapper/220:25:132
1473205271,2sleep10-21swapper/120:15:181
1470714271,2sleep10-21swapper/120:10:211
1457853271,2sleep00-21swapper/020:00:180
1455351270,2sleep21455520-21strings19:55:182
1450974270,1sleep20-21swapper/219:50:552
1406123270,1sleep20-21swapper/219:11:072
1525830261,2sleep20-21swapper/221:05:132
1505236261,2sleep00-21swapper/020:45:190
1488515260,1sleep00-21swapper/020:30:150
1488021260,1sleep10-21swapper/120:27:091
1482715260,4sleep10-21swapper/120:22:201
1472670261,2sleep01472687-21apt20:15:130
1471311260,2sleep20-21swapper/220:10:222
1435162260,1sleep20-21swapper/219:39:352
1431419261,2sleep10-21swapper/119:35:181
1420204261,2sleep10-21swapper/119:25:161
1419636260,2sleep00-21swapper/019:25:110
1514962250,1sleep00-21swapper/020:55:010
1481991250,1sleep00-21swapper/020:20:210
1509760240,1sleep20-21swapper/220:50:022
1461915240,1sleep20-21swapper/220:05:002
1446252240,1sleep10-21swapper/119:50:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional