You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 18:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 10, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
175558221450,4sleep01755585-21threads10:45:290
1482677211755,57sleep30-21swapper/307:05:443
148295528957,6sleep00-21swapper/007:09:130
148267528756,9sleep10-21swapper/107:05:421
148296228455,9sleep20-21swapper/207:09:192
1483141992926,1cyclictest0-21swapper/309:40:003
15986402279,14sleep21600449-21cut08:55:202
148314199258,8cyclictest1570232-21turbostat08:29:593
1483141992521,2cyclictest0-21swapper/310:10:013
1483141992414,4cyclictest17057502sleep310:12:093
148314199232,15cyclictest1845799-21taskset11:50:133
1483141992319,2cyclictest0-21swapper/311:40:293
1483141992316,4cyclictest1564944-21turbostat08:25:003
148314199230,18cyclictest1624801-21taskset09:15:443
148314199222,13cyclictest1842663-21taskset11:45:253
1483141992219,1cyclictest0-21swapper/309:45:213
1483141992218,2cyclictest0-21swapper/312:15:183
1483141992117,2cyclictest0-21swapper/310:40:193
1483141992116,2cyclictest0-21swapper/311:25:093
148314199211,16cyclictest1685026-21taskset09:59:143
148314199211,15cyclictest0-21swapper/309:10:113
148314199211,15cyclictest0-21swapper/307:29:313
17404702201,14sleep11740511-21collect210:35:211
17404702201,14sleep11740511-21collect210:35:201
148314199202,13cyclictest1805593-21taskset11:21:393
148314199202,12cyclictest1897665-21taskset12:25:063
1483141992017,1cyclictest0-21swapper/309:30:553
1483141992017,1cyclictest0-21swapper/307:37:493
1483141992017,1cyclictest0-21swapper/307:37:483
1483141992016,2cyclictest0-21swapper/312:34:223
1483141992016,2cyclictest0-21swapper/311:00:003
148314199201,15cyclictest1690948-21taskset10:01:133
148314199201,15cyclictest0-21swapper/311:55:143
148314199200,18cyclictest0-21swapper/311:17:303
148314199200,13cyclictest1714165-21taskset10:19:263
16325222191,12sleep11612506-21diskmemload09:22:321
148314199193,11cyclictest1757900-21taskset10:50:053
148314199192,12cyclictest1865510-21taskset12:00:393
1483141991916,1cyclictest0-21swapper/311:32:243
1483141991915,2cyclictest0-21swapper/311:11:333
1483141991915,2cyclictest0-21swapper/307:14:593
1483141991914,3cyclictest66550irq/131-enp1s0-TxRx-309:28:383
1483141991914,2cyclictest0-21swapper/309:42:333
1483141991913,3cyclictest0-21swapper/310:20:153
148314199191,14cyclictest1677746-21chrt09:54:123
148314199191,14cyclictest0-21swapper/312:20:163
148314199191,12cyclictest1726810-21taskset10:25:323
18362472181,10sleep20-21swapper/211:41:512
148314199187,8cyclictest1735528-21chrt10:34:083
148314199182,12cyclictest1632648-21taskset09:22:533
148314199182,11cyclictest1913108-21taskset12:35:123
1483141991815,2cyclictest0-21swapper/307:50:183
1483141991815,1cyclictest0-21swapper/308:15:253
148314199181,4cyclictest0-21swapper/307:20:143
1483141991814,2cyclictest0-21swapper/310:35:153
1483141991814,2cyclictest0-21swapper/310:35:143
1483141991814,2cyclictest0-21swapper/309:10:013
1483141991814,2cyclictest0-21swapper/307:45:183
1483141991812,3cyclictest0-21swapper/311:35:163
148314199181,13cyclictest1756048-21chrt10:45:553
19104222171,10sleep01612506-21diskmemload12:30:530
18211872171,4sleep11821199-21sh11:32:201
17708842171,4sleep01612506-21diskmemload10:59:570
17339122171,3sleep00-21swapper/010:30:430
17203522171,3sleep20-21swapper/210:22:312
16983012171,5sleep01612506-21diskmemload10:06:490
16687842171,3sleep10-21swapper/109:45:581
16624182171,3sleep00-21swapper/009:43:160
16482912171,11sleep21648351-21munin-run09:35:002
15320322171,11sleep11532090-21grep07:50:221
148314199179,4cyclictest18810942sleep312:12:353
148314199179,4cyclictest15916272sleep308:50:113
1483141991714,1cyclictest0-21swapper/308:00:173
1483141991714,1cyclictest0-21swapper/307:30:193
1483141991713,2cyclictest0-21swapper/311:05:163
1483141991713,2cyclictest0-21swapper/308:45:013
148314199171,13cyclictest0-21swapper/312:05:153
148314199171,12cyclictest1776347-21chrt11:01:073
148314199171,12cyclictest1522494-21turbostat07:45:013
19036482161,9sleep21612506-21diskmemload12:27:262
18958082161,9sleep11612506-21diskmemload12:22:081
18294782161,9sleep11612506-21diskmemload11:39:031
18289992161,1sleep00-21swapper/011:38:040
18264412161,9sleep21826932-21/usr/sbin/munin11:35:192
17500972161,2sleep00-21swapper/010:43:570
17441192161,11sleep21745552-21kthreadcore10:40:182
17405122161,4sleep21740560-21ntpq10:35:222
17405122161,4sleep21740560-21ntpq10:35:212
15536482161,10sleep11553699-21sed08:10:261
148314199161,9cyclictest1544016-21taskset08:05:013
148314199161,3cyclictest0-21swapper/308:55:143
1483141991613,1cyclictest0-21swapper/308:45:163
148314199160,13cyclictest0-21swapper/307:15:173
148314199160,10cyclictest1554333-21taskset08:14:563
19047462150,3sleep11612506-21diskmemload12:29:291
18963082151,8sleep20-21swapper/212:23:082
18456812158,1sleep20-21swapper/211:50:052
17650642151,9sleep11765355-21cut10:55:171
16063442151,9sleep11606454-21uname09:00:231
1483141991512,1cyclictest0-21swapper/308:35:163
148314199150,12cyclictest0-21swapper/309:00:123
148314199150,12cyclictest0-21swapper/307:55:223
19186502141,4sleep21918660-21ssh12:36:472
18573082141,7sleep11857355-21ntpq11:55:211
18445912140,2sleep00-21swapper/011:48:190
18438962141,2sleep20-21swapper/211:47:012
17841872140,3sleep10-21swapper/111:08:081
17622262141,8sleep11762263-21ssh10:50:231
17057392140,7sleep244-21ksoftirqd/210:12:082
17053382141,7sleep11612506-21diskmemload10:11:201
16712312141,9sleep01671287-21ls09:50:140
16559102141,6sleep11655982-21/usr/sbin/munin09:40:011
15586632142,3sleep21558717-21ntpq08:15:242
15214792141,9sleep20-21swapper/207:40:222
15113652140,1sleep20-21swapper/207:30:352
15106752141,9sleep11510728-21ld07:30:221
148314199141,10cyclictest1574502-21chrt08:30:193
18814862131,8sleep01881496-21ssh12:13:290
18674472131,7sleep01867453-21ssh12:04:510
18069092131,7sleep01806922-21ssh11:24:010
17922902131,8sleep11792303-21ssh11:14:281
17781052130,2sleep11612506-21diskmemload11:04:581
17356572131,3sleep11735662-21ssh10:34:251
17280242131,6sleep21612506-21diskmemload10:28:312
17271932131,8sleep01727202-21ssh10:26:490
17002042131,5sleep01700466-21df10:10:140
16918152130,9sleep21691826-21ssh10:03:302
16850762131,8sleep01685085-21ssh09:59:190
15285952132,7sleep01529630-21grep07:50:170
18203922121,4sleep21612506-21diskmemload11:30:532
18066782121,4sleep21612506-21diskmemload11:23:352
17908892121,4sleep01612506-21diskmemload11:11:470
17845822121,8sleep00-21swapper/011:09:070
17558592121,7sleep21755865-21ssh10:45:322
17187142121,2sleep00-21swapper/010:20:200
16979842121,3sleep21697989-21ssh10:06:112
16704032121,4sleep01612506-21diskmemload09:49:050
16657242121,7sleep21667505-21kthreadcore09:45:192
16404022121,3sleep10-21swapper/109:28:581
15748172121,7sleep01574845-21switchtime08:30:240
15490272121,7sleep01549203-21latency_hist08:10:010
14909272124,1sleep2171rcu_preempt07:15:172
19200372111,4sleep11612506-21diskmemload12:39:151
19104302111,4sleep11612506-21diskmemload12:30:541
18892472111,5sleep11612506-21diskmemload12:18:581
18819052111,3sleep10-21swapper/112:14:261
18727092111,4sleep11612506-21diskmemload12:05:311
18431122111,4sleep11612506-21diskmemload11:45:361
18142622111,2sleep00-21swapper/011:28:230
18118532111,5sleep21811908-21ntpq11:25:212
17940352111,6sleep21795622-21kthreadcore11:15:182
17882602111,4sleep21789309-21memory11:10:162
17639362111,3sleep00-21swapper/010:54:070
17574942111,4sleep11612506-21diskmemload10:49:441
17280512111,4sleep11612506-21diskmemload10:28:341
17199422111,4sleep11612506-21diskmemload10:21:301
16978232111,4sleep11612506-21diskmemload10:05:531
16917902111,1sleep10-21swapper/110:03:261
16863712111,6sleep01687045-21cut10:00:160
16841852111,4sleep21612506-21diskmemload09:57:212
16839982111,3sleep10-21swapper/109:56:521
16771782111,4sleep11612506-21diskmemload09:53:031
16624002110,5sleep20-21swapper/209:43:132
16567942111,6sleep11657239-21kthreadcore09:40:161
16529652111,5sleep21653020-21ntpq09:35:212
16474922111,4sleep11612506-21diskmemload09:33:141
16338762111,5sleep21633900-21apt-get09:25:112
16188312111,4sleep21612506-21diskmemload09:13:352
16170592111,4sleep11612506-21diskmemload09:10:211
15813732111,6sleep21581520-21tail08:40:162
15677472112,4sleep21569064-21memory08:25:172
15507762111,6sleep01552540-21kthreadcore08:10:170
15171472111,5sleep11517207-21modprobe07:40:011
15171472111,5sleep11517207-21modprobe07:40:001
15169662110,1sleep20-21swapper/207:37:532
15169662110,1sleep20-21swapper/207:37:522
15169542111,5sleep01516921-31find07:37:510
15169542111,5sleep01516921-31find07:37:500
19182432101,5sleep01918256-21ssh12:36:050
19105792101,5sleep21910584-21ssh12:31:102
19030862101,3sleep00-21swapper/012:26:230
18954342101,3sleep00-21swapper/012:21:190
18892582101,3sleep20-21swapper/212:19:002
18825612101,4sleep01882566-21apt-get12:15:120
18803002101,2sleep20-21swapper/212:10:452
18640592101,2sleep20-21swapper/212:00:192
18584392101,3sleep21612506-21diskmemload11:56:272
18572452100,1sleep00-21swapper/011:55:200
18477982100,6sleep01849550-21kthreadcore11:50:200
18457562100,4sleep11612506-21diskmemload11:50:131
18225312101,4sleep01822588-21munin-run11:35:010
18120362101,5sleep11812158-21ps11:25:241
18060142101,4sleep11612506-21diskmemload11:22:261
17995632101,2sleep10-21swapper/111:19:141
17984712100,3sleep00-21swapper/011:17:090
17790622101,5sleep21780217-21tr11:05:162
17708982100,4sleep21612506-21diskmemload10:59:592
17484722101,3sleep11612506-21diskmemload10:40:571
17289462101,4sleep21728967-21apt-get10:30:132
17129332101,3sleep21612506-21diskmemload10:17:022
17073842101,5sleep01707447-21ls10:15:140
16780752101,3sleep20-21swapper/209:54:562
16488252101,3sleep01648310-21apt-get09:35:020
16461992101,3sleep01612506-21diskmemload09:30:370
16335292101,4sleep01633587-21munin-run09:24:590
16257822101,2sleep00-21swapper/009:18:230
16199022101,3sleep21612506-21diskmemload09:15:122
16080632101,4sleep01609278-21cut09:05:160
16008262101,5sleep01600915-21ntpq08:55:220
15866372101,5sleep11586824-21head08:45:171
15801822101,4sleep11580426-21users08:35:271
15707662104,3sleep11570890-21grep08:30:131
15599332101,5sleep21560124-21expr08:20:132
15585012101,4sleep01558614-21ntpq08:15:230
15374482101,4sleep01537501-21ntp_states07:55:230
15336272101,4sleep11533836-21hddtemp_smartct07:55:151
14906012101,4sleep01490856-21df07:15:150
1874875291,2sleep00-21swapper/012:09:470
1873493291,2sleep21612506-21diskmemload12:07:102
1866601291,2sleep10-21swapper/112:03:041
1836130291,1sleep10-21swapper/111:41:381
1830169291,3sleep01830026-21apt-get11:40:020
1776424290,1sleep00-21swapper/011:01:160
1776033290,1sleep20-21swapper/211:00:262
1757893290,3sleep21612506-21diskmemload10:50:042
1741348291,2sleep00-21swapper/010:36:000
1741348291,2sleep00-21swapper/010:35:590
1633814291,3sleep01612506-21diskmemload09:25:030
1625863291,4sleep11625873-21ssh09:18:321
1618190291,2sleep00-21swapper/009:12:210
1602200291,3sleep21602222-21apt-get09:00:122
1597467291,4sleep11598639-21kthreadcore08:55:191
1591923291,3sleep21592137-21smartctl08:50:142
1580627290,1sleep00-21swapper/008:37:550
1575509291,4sleep21575541-21perf08:34:592
1565725291,4sleep01566041-21grep08:25:160
1548144290,5sleep21548258-21/usr/sbin/munin08:05:222
1517719290,1sleep00-21swapper/007:40:150
1494920291,4sleep11495023-21/usr/sbin/munin07:15:281
1487676291,4sleep21487928-21grep07:10:272
1712185280,2sleep11612506-21diskmemload10:15:311
1629671281,3sleep21630809-21memory09:20:162
1609279281,3sleep11611088-21kthreadcore09:05:171
1591650281,3sleep01591812-21wc08:50:120
1588073281,3sleep01589857-21cut08:45:200
1564934280,1sleep00-21swapper/008:24:590
1506069280,1sleep20-21swapper/207:28:432
1498857281,3sleep11499777-21sed07:20:201
1612215270,1sleep20-21swapper/209:06:312
1585138271,3sleep11585199-21ntp_offset08:40:231
1559911271,2sleep11559930-21apt-get08:20:111
1555423271,2sleep10-21swapper/108:15:191
1542900270,4sleep00-21swapper/008:00:230
1538719271,2sleep11538737-21apt-get08:00:121
1527441270,1sleep00-21swapper/007:45:570
1511624270,1sleep00-21swapper/007:33:500
1501734271,3sleep10-21swapper/107:25:181
1488158271,3sleep172-21kcompactd007:12:501
1606537260,3sleep00-21swapper/009:00:260
1595941261,2sleep10-21swapper/108:50:251
1591219260,1sleep20-21swapper/208:48:112
1581106261,2sleep00-21swapper/008:40:140
1554183262,2sleep2431ktimers/208:13:042
1532900260,1sleep20-21swapper/207:52:522
1527565260,1sleep20-21swapper/207:47:302
1527465260,1sleep10-21swapper/107:46:131
1497109261,2sleep00-21swapper/007:20:190
1488044262,2sleep0151ktimers/007:11:250
1575773250,1sleep20-21swapper/208:35:122
1566043250,1sleep10-21swapper/108:25:171
1543765250,2sleep10-21swapper/108:05:011
1542795250,1sleep20-21swapper/208:00:222
1533337250,1sleep20-21swapper/207:55:122
1521384250,1sleep10-21swapper/107:40:201
1501579251,1sleep00-21swapper/007:25:160
1496246250,1sleep20-21swapper/207:20:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional