You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 03:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sat Dec 06, 2025 00:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1622129212956,64sleep30-21swapper/319:05:483
197510121080,3sleep1321rcuc/123:25:211
185741421010,2sleep30-21swapper/322:11:383
162229229261,13sleep10-21swapper/119:07:501
162242927956,6sleep00-21swapper/019:09:380
162223027761,10sleep20-21swapper/219:07:032
1622588993329,2cyclictest0-21swapper/322:20:013
1622588992724,1cyclictest0-21swapper/320:50:193
19440442251,14sleep21944219-21grep23:05:252
162258899242,13cyclictest2010651-21taskset23:49:563
1622588992421,1cyclictest0-21swapper/323:25:003
1622588992319,2cyclictest0-21swapper/322:55:173
1622588992317,3cyclictest0-21swapper/321:55:153
162258899231,13cyclictest1776484-21taskset21:24:563
20318642221,2sleep00-21swapper/000:00:440
1622588992219,1cyclictest0-21swapper/321:30:173
1622588992218,2cyclictest0-21swapper/322:30:163
1622588992218,2cyclictest0-21swapper/321:50:173
1622588992217,2cyclictest0-21swapper/321:43:543
162258899221,17cyclictest2063491-21taskset00:21:433
162258899221,17cyclictest2063491-21taskset00:21:433
162258899221,16cyclictest1810816-21taskset21:45:123
162258899212,12cyclictest2082574-21taskset00:35:183
1622588992119,1cyclictest0-21swapper/320:40:003
1622588992118,1cyclictest0-21swapper/322:35:213
1622588992117,2cyclictest0-21swapper/320:19:593
1622588992111,7cyclictest1640275-21chrt19:23:273
162258899211,15cyclictest1979316-21chrt23:30:123
162258899211,15cyclictest1777134-21chrt21:25:083
162258899210,17cyclictest1645313-21taskset19:25:223
20161462201,4sleep21752182-21diskmemload23:50:522
162258899202,12cyclictest2018905-21taskset23:55:093
1622588992017,1cyclictest0-21swapper/321:40:003
1622588992016,2cyclictest0-21swapper/323:52:423
1622588992014,3cyclictest0-21swapper/300:10:173
162258899201,15cyclictest2000313-21taskset23:40:543
162258899201,15cyclictest1841715-21taskset22:02:523
162258899201,14cyclictest2031727-21chrt00:00:303
162258899201,14cyclictest1913401-21chrt22:46:503
162258899201,14cyclictest1913401-21chrt22:46:493
162258899192,12cyclictest1961720-21taskset23:17:533
162258899192,11cyclictest2076641-21taskset00:30:213
1622588991916,1cyclictest0-21swapper/319:50:163
1622588991915,3cyclictest0-21swapper/323:25:253
1622588991915,2cyclictest0-21swapper/322:20:183
1622588991914,3cyclictest66550irq/131-enp1s0-TxRx-323:10:073
1622588991914,2cyclictest0-21swapper/322:25:093
162258899191,14cyclictest1993966-21chrt23:38:333
162258899191,14cyclictest1848691-21chrt22:05:573
162258899191,14cyclictest1757459-21chrt21:10:363
162258899191,14cyclictest0-21swapper/323:00:473
162258899191,14cyclictest0-21swapper/322:50:113
162258899191,14cyclictest0-21swapper/321:15:073
162258899191,14cyclictest0-21swapper/300:05:093
162258899191,13cyclictest1622779-21chrt19:10:123
20801512181,2sleep10-21swapper/100:35:171
20336612181,4sleep21752182-21diskmemload00:03:472
20084622181,6sleep01752182-21diskmemload23:46:250
19369182181,3sleep10-21swapper/123:00:561
17656762181,4sleep21752182-21diskmemload21:15:542
1622588991815,2cyclictest0-21swapper/320:45:153
162258899181,4cyclictest1720682-21taskset20:40:143
1622588991813,3cyclictest0-21swapper/300:25:153
1622588991813,3cyclictest0-21swapper/300:25:153
162258899181,13cyclictest2054664-21chrt00:15:263
162258899181,13cyclictest0-21swapper/323:05:103
162258899181,13cyclictest0-21swapper/322:40:363
162258899181,13cyclictest0-21swapper/320:00:203
20641702171,4sleep21752182-21diskmemload00:22:552
20641702171,4sleep21752182-21diskmemload00:22:552
19703632171,11sleep01970373-21ssh23:24:120
19378892171,2sleep00-21swapper/023:02:400
19076862171,4sleep21752182-21diskmemload22:45:102
19076862171,4sleep21752182-21diskmemload22:45:102
17914872175,3sleep21752182-21diskmemload21:31:572
17836562171,11sleep21783665-21ssh21:27:562
17834532171,3sleep01752182-21diskmemload21:27:380
17267892176,7sleep11728488-21cut20:45:141
1622588991714,1cyclictest0-21swapper/320:35:013
1622588991711,4cyclictest0-21swapper/319:15:123
162258899171,13cyclictest1752207-21sleep21:10:003
162258899171,13cyclictest0-21swapper/321:00:143
162258899171,13cyclictest0-21swapper/321:00:143
162258899171,13cyclictest0-21swapper/320:05:183
162258899171,13cyclictest0-21swapper/319:30:123
162258899171,12cyclictest1661881-21turbostat19:45:003
162258899171,12cyclictest0-21swapper/320:55:133
20472182160,2sleep20-21swapper/200:10:372
19942682161,10sleep11752182-21diskmemload23:39:041
19057662161,2sleep20-21swapper/222:42:332
18582012161,3sleep00-21swapper/022:12:480
18514202161,10sleep11851551-21date22:10:011
18105992163,7sleep11810627-21apt-get21:45:111
1622588991614,1cyclictest0-21swapper/319:55:113
162258899160,13cyclictest0-21swapper/320:25:263
162258899160,13cyclictest0-21swapper/319:45:123
19692572151,2sleep20-21swapper/223:22:182
19682832154,3sleep11968294-21diskmemload23:20:371
19159392150,2sleep10-21swapper/122:50:101
19150592151,2sleep10-21swapper/122:49:341
19150592151,2sleep10-21swapper/122:49:341
18339022151,2sleep10-21swapper/121:58:381
18248982150,3sleep10-21swapper/121:52:201
18166742151,7sleep00-21swapper/021:47:150
18162582151,2sleep20-21swapper/221:46:372
18092852151,3sleep21752182-21diskmemload21:43:322
17919152151,9sleep01791926-21ssh21:32:340
17764712151,3sleep21752182-21diskmemload21:24:562
17209632151,9sleep11722151-21cut20:40:171
17148012151,3sleep01714820-21apt-get20:35:110
16393002151,4sleep21639388-21ntpq19:20:182
162258899156,4cyclictest16551432sleep319:35:173
162258899151,9cyclictest1703683-21chrt20:20:463
162258899151,10cyclictest1693490-21turbostat20:15:003
20550862140,8sleep014-21ksoftirqd/000:15:360
20256342145,2sleep00-21swapper/023:58:400
20181932141,7sleep01752182-21diskmemload23:54:230
19952022142,4sleep00-21swapper/023:40:120
18735412141,9sleep01873542-21sh22:21:340
18673232141,9sleep01867330-21ssh22:19:300
18394862141,9sleep11839563-21ls22:00:221
17752652141,7sleep01752182-21diskmemload21:23:010
17461892141,9sleep11746194-21sendmail_mailtr21:00:251
17461892141,9sleep11746194-21sendmail_mailtr21:00:241
17090332140,1sleep10-21swapper/120:27:031
20416742131,7sleep22041688-21ssh00:09:322
19935512131,5sleep00-21swapper/023:37:550
19607672130,2sleep00-21swapper/023:16:150
19542362130,7sleep21752182-21diskmemload23:13:362
19311712131,8sleep11931178-21ssh22:59:281
19281662131,2sleep00-21swapper/022:55:220
19159742131,5sleep01752182-21diskmemload22:50:130
18978042131,2sleep01897815-21ssh22:37:420
18964632131,2sleep10-21swapper/122:35:301
18897262131,5sleep01752182-21diskmemload22:32:290
18744602130,8sleep11752182-21diskmemload22:23:001
18729592131,8sleep21872970-21ssh22:20:352
18329982130,2sleep0171rcu_preempt21:57:170
17908502131,5sleep10-21swapper/121:30:591
17669572131,7sleep01752182-21diskmemload21:17:520
17528472131,8sleep01753048-21sed21:10:140
17262282131,7sleep01726786-21kthreadcore20:45:130
17002952131,7sleep11702021-21grep20:20:171
16232902131,7sleep21623584-21kthreadcore19:10:162
20573682121,8sleep10-21swapper/100:19:371
20479632120,1sleep00-21swapper/000:11:510
20476402120,1sleep10-21swapper/100:11:201
20173882121,4sleep11752182-21diskmemload23:53:001
20086062121,7sleep12008618-21ssh23:46:371
19851952120,3sleep01985206-21ssh23:32:140
19847712120,1sleep20-21swapper/223:31:322
19762082121,5sleep21752182-21diskmemload23:25:542
19609152121,6sleep11960926-21ssh23:16:281
19539332121,5sleep00-21swapper/023:13:060
19518422120,1sleep10-21swapper/123:10:221
19368562121,4sleep21752182-21diskmemload23:00:492
19299892121,5sleep21752182-21diskmemload22:57:232
19118592121,7sleep01911910-21collect222:45:200
19118592121,7sleep01911910-21collect222:45:200
19071442120,6sleep10-21swapper/122:44:451
18968672121,5sleep21752182-21diskmemload22:36:122
18894482121,5sleep20-21swapper/222:31:582
18600532121,5sleep21860065-21apt-get22:15:152
18579342120,8sleep11857943-21ssh22:12:251
18480202121,7sleep01848181-21cat22:05:270
18353082121,5sleep01835522-21diskstats22:00:140
18262482121,5sleep00-21swapper/021:54:230
18243882121,4sleep21752182-21diskmemload21:51:342
18005672120,8sleep11800575-21sh21:38:141
18002982121,5sleep21752182-21diskmemload21:37:512
17664902121,5sleep11752182-21diskmemload21:17:091
17326112121,6sleep01734374-21kthreadcore20:50:180
17307392120,7sleep21730878-21sort20:49:592
16977302121,5sleep11697776-21ntpq20:15:201
16949502121,6sleep21696667-21kthreadcore20:15:162
16512922120,1sleep00-21swapper/019:34:510
16300452120,1sleep00-21swapper/019:15:110
20837052111,3sleep20-21swapper/200:35:212
20786252111,3sleep10-21swapper/100:33:371
20782602111,4sleep21752182-21diskmemload00:32:452
20717592110,3sleep10-21swapper/100:28:521
20717592110,3sleep10-21swapper/100:28:521
20710212111,3sleep00-21swapper/000:26:500
20710212111,3sleep00-21swapper/000:26:500
20655462111,4sleep21752182-21diskmemload00:25:082
20655462111,4sleep21752182-21diskmemload00:25:072
20637852111,3sleep10-21swapper/100:22:141
20637852111,3sleep10-21swapper/100:22:131
20630442111,4sleep01752182-21diskmemload00:20:570
20630442111,4sleep01752182-21diskmemload00:20:570
20565412111,2sleep22056549-21ssh00:18:102
20404132111,4sleep01752182-21diskmemload00:07:180
20394922111,3sleep10-21swapper/100:05:371
20268282111,4sleep11752182-21diskmemload00:00:131
20258592111,4sleep21752182-21diskmemload23:59:022
20102372111,4sleep21752182-21diskmemload23:49:162
20016852111,3sleep20-21swapper/223:43:272
20002912111,4sleep10-21swapper/123:40:531
19872772111,6sleep21987377-21cpuspeed_turbos23:35:132
19863642111,4sleep11752182-21diskmemload23:34:071
19776202111,3sleep01752182-21diskmemload23:28:170
19610222111,3sleep20-21swapper/223:16:402
19457662111,3sleep10-21swapper/123:07:341
19215522111,4sleep21752182-21diskmemload22:51:392
18837742111,5sleep10-21swapper/122:30:131
18832342111,4sleep21752182-21diskmemload22:29:422
18829632111,3sleep01752182-21diskmemload22:29:120
18759362111,4sleep11752182-21diskmemload22:25:081
18574852111,3sleep21752182-21diskmemload22:11:452
18509272111,4sleep21752182-21diskmemload22:09:182
18405082111,4sleep21752182-21diskmemload22:01:032
18321052111,4sleep21752182-21diskmemload21:55:552
18075622111,3sleep10-21swapper/121:40:541
18073332111,4sleep01752182-21diskmemload21:40:320
17980032111,5sleep01798061-21ntpq21:35:200
17747712111,3sleep10-21swapper/121:22:131
17576952111,3sleep11752182-21diskmemload21:10:591
17531472111,6sleep21754228-21kthreadcore21:10:152
17412882111,3sleep01741445-21sed21:00:000
16822612111,6sleep21682371-21wc20:00:262
16729562110,7sleep21672999-21fschecks_count19:55:132
20854382101,3sleep01752182-21diskmemload00:38:090
20776392101,2sleep00-21swapper/000:31:110
20188402101,3sleep10-21swapper/123:55:031
19451872101,3sleep00-21swapper/023:06:400
19047322101,2sleep00-21swapper/022:40:510
18654342101,3sleep10-21swapper/122:16:261
17771162101,3sleep10-21swapper/121:25:071
17418722101,4sleep21742127-21df21:00:142
17418722101,4sleep21742127-21df21:00:132
16990312101,3sleep01699050-21apt-get20:20:120
16837302101,5sleep11684003-21cut20:05:141
16819282101,4sleep01681973-21ntpq20:00:190
16726742101,3sleep00-21swapper/019:55:110
16568592101,4sleep01656878-21apt-get19:40:120
16564232100,1sleep10-21swapper/119:38:051
16467672101,4sleep11646838-21irqstats19:30:151
1735110291,4sleep11735198-21ps20:50:231
1722153291,4sleep21723914-21kthreadcore20:40:182
1720864291,4sleep01720962-21irqstats20:40:160
1718892291,4sleep21718948-21ntp_states20:35:202
1710937291,4sleep11712665-21grep20:30:191
1705991291,4sleep21707773-21cut20:25:182
1703103291,4sleep21703150-21/usr/sbin/munin20:20:222
1688512290,3sleep10-21swapper/120:10:151
1676711291,3sleep11676760-21ntpq19:55:201
1667944291,4sleep21668878-21kthreadcore19:50:162
1667116291,3sleep21667233-21cat19:50:002
1637063291,4sleep01638818-21grep19:20:160
1752024280,1sleep10-21swapper/121:08:071
1736264280,3sleep11736283-21apt-get20:55:121
1719835281,3sleep11720281-21apt-key20:40:011
1709532281,2sleep00-21swapper/020:30:140
1687269281,3sleep21687316-21ntpq20:05:182
1667065280,1sleep10-21swapper/119:49:231
1658113281,3sleep11659831-21kthreadcore19:40:171
1653361281,3sleep21655140-21kthreadcore19:35:162
1646839281,3sleep20-21swapper/219:30:162
1645002282,3sleep21622582-21cyclictest19:25:182
1640934280,1sleep10-21swapper/119:25:101
1634115280,4sleep11634168-21modprobe19:15:191
1627798281,3sleep11627821-21munin-run19:15:001
1751822270,4sleep20-21swapper/221:05:352
1747174270,2sleep00-21swapper/021:05:130
1746859270,1sleep00-21swapper/021:04:210
1746859270,1sleep00-21swapper/021:04:210
1741172270,1sleep20-21swapper/220:58:332
1734906270,3sleep21735001-21modprobe20:50:202
1709090270,1sleep00-21swapper/020:27:460
1688534271,2sleep20-21swapper/220:10:162
1678744271,2sleep10-21swapper/120:00:171
1671283271,2sleep10-21swapper/119:50:191
1662924270,3sleep01664574-21cut19:45:170
1645113271,2sleep01645198-21ps19:25:210
1627835272,2sleep20-21swapper/219:15:002
1698667260,1sleep00-21swapper/020:19:000
1693205260,1sleep00-21swapper/020:11:390
1661670260,1sleep20-21swapper/219:42:312
1640153260,1sleep10-21swapper/119:21:541
1626875261,2sleep00-21swapper/019:10:220
1713855250,1sleep20-21swapper/220:30:282
1683004250,1sleep00-21swapper/020:05:010
1651564250,1sleep00-21swapper/019:35:110
1667663240,1sleep00-21swapper/019:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional