You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 06:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 08, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2021118211756,56sleep30-21swapper/319:07:313
202109428654,9sleep20-21swapper/219:07:142
202101128560,19sleep10-21swapper/119:06:141
202104828358,19sleep00-21swapper/019:06:410
2021443992722,3cyclictest0-21swapper/323:35:143
202144399262,18cyclictest2466157-21chrt00:25:143
23614402250,3sleep10-21swapper/123:17:011
202144399242,14cyclictest2476294-21taskset00:30:163
202144399242,14cyclictest2215924-21taskset21:50:123
2021443992421,1cyclictest0-21swapper/323:25:223
2021443992420,2cyclictest0-21swapper/323:45:113
202144399240,21cyclictest66550irq/131-enp1s0-TxRx-300:10:243
2021443992319,2cyclictest0-21swapper/321:15:173
202144399231,18cyclictest2263833-21taskset22:19:193
202144399231,18cyclictest2263833-21taskset22:19:183
202144399231,14cyclictest2213979-21taskset21:47:353
23208232221,2sleep20-21swapper/222:53:062
23137142221,5sleep12150347-21diskmemload22:49:401
202144399222,14cyclictest2385488-21taskset23:31:353
202144399222,14cyclictest2190048-21taskset21:33:373
2021443992219,1cyclictest0-21swapper/320:40:183
2021443992219,1cyclictest0-21swapper/320:14:593
2021443992214,4cyclictest21734112sleep321:22:593
202144399221,14cyclictest0-21swapper/322:55:213
24511602211,2sleep10-21swapper/100:15:001
2021443992119,1cyclictest0-21swapper/300:39:593
2021443992118,1cyclictest0-21swapper/322:30:353
2021443992117,3cyclictest0-21swapper/323:05:013
2021443992117,2cyclictest0-21swapper/323:05:283
2021443992117,2cyclictest0-21swapper/320:40:003
2021443992111,6cyclictest66550irq/131-enp1s0-TxRx-323:54:463
202144399210,19cyclictest0-21swapper/323:15:433
23528162201,9sleep12352821-21ssh23:11:091
23196282200,9sleep02319638-21ssh22:51:120
22325252201,14sleep02232718-21tail22:00:160
2021443992017,1cyclictest0-21swapper/322:50:173
2021443992017,1cyclictest0-21swapper/322:30:013
2021443992017,1cyclictest0-21swapper/320:05:263
2021443992016,2cyclictest0-21swapper/322:45:143
2021443992016,2cyclictest0-21swapper/321:10:193
2021443992016,2cyclictest0-21swapper/320:50:003
2021443992016,2cyclictest0-21swapper/300:20:173
2021443992015,2cyclictest0-21swapper/321:26:313
202144399201,16cyclictest0-21swapper/321:58:443
202144399201,15cyclictest0-21swapper/322:35:133
202144399201,14cyclictest0-21swapper/322:05:133
2021443992010,7cyclictest2435653-21taskset00:04:263
202144399200,1cyclictest0-21swapper/322:44:443
23148732191,13sleep12315464-21grep22:50:171
21989302191,2sleep20-21swapper/221:39:532
20772052191,7sleep12077565-21kthreadcore20:00:161
202144399192,12cyclictest2442383-21taskset00:07:463
2021443991916,1cyclictest0-21swapper/320:50:163
2021443991915,2cyclictest0-21swapper/322:00:223
2021443991915,2cyclictest0-21swapper/321:40:173
202144399191,14cyclictest0-21swapper/322:10:123
202144399191,13cyclictest2363748-21taskset23:20:123
24101112181,4sleep12150347-21diskmemload23:46:531
23376382181,8sleep12150347-21diskmemload23:03:121
23052042181,4sleep12150347-21diskmemload22:44:031
22700752181,5sleep22150347-21diskmemload22:21:192
22219002181,10sleep12150347-21diskmemload21:52:161
22084202181,11sleep12209226-21tr21:45:181
202144399182,12cyclictest2097513-21turbostat20:20:003
202144399182,11cyclictest2424507-21taskset23:55:193
2021443991815,1cyclictest0-21swapper/319:20:173
2021443991814,2cyclictest0-21swapper/319:50:013
202144399181,14cyclictest0-21swapper/323:10:293
202144399181,14cyclictest0-21swapper/321:36:053
202144399181,13cyclictest0-21swapper/322:20:073
202144399181,11cyclictest2452051-21taskset00:15:163
202144399181,11cyclictest2107839-21chrt20:26:353
202144399181,11cyclictest2107839-21chrt20:26:343
2021443991810,4cyclictest24014552sleep323:41:023
24340562171,3sleep20-21swapper/200:01:302
23710032171,3sleep20-21swapper/223:24:172
22628632171,5sleep02262874-21sh22:17:480
22628632171,5sleep02262874-21sh22:17:470
22548012171,4sleep12150347-21diskmemload22:12:551
22132812171,3sleep20-21swapper/221:46:302
21990162171,10sleep12199095-21munin-run21:40:011
21573342171,3sleep10-21swapper/121:13:151
21190082171,10sleep22119254-21df_inode20:40:172
202144399173,9cyclictest2026221-21taskset19:10:363
202144399173,10cyclictest2150370-21sleep21:09:593
2021443991714,1cyclictest0-21swapper/319:42:143
2021443991713,2cyclictest0-21swapper/319:40:013
2021443991713,2cyclictest0-21swapper/319:15:173
202144399171,12cyclictest2113441-21sleep20:35:013
24592802161,9sleep02459820-21kthreadcore00:20:150
24569732161,9sleep12150347-21diskmemload00:17:001
24495922161,9sleep02150347-21diskmemload00:12:010
24261512161,3sleep10-21swapper/123:56:521
24188172161,4sleep12150347-21diskmemload23:52:581
23636092161,2sleep00-21swapper/023:20:020
23466402161,8sleep00-21swapper/023:09:390
23279822161,8sleep12150347-21diskmemload22:56:051
22867502161,2sleep10-21swapper/122:31:201
22780582161,4sleep22150347-21diskmemload22:25:592
22543352161,2sleep00-21swapper/022:12:100
22408752161,9sleep02240955-21irqstats22:05:180
21746942162,7sleep02174749-21munin-run21:25:000
202144399161,9cyclictest2039522-21taskset19:25:073
2021443991612,2cyclictest0-21swapper/320:55:193
202144399160,13cyclictest0-21swapper/320:03:433
24737632151,9sleep22474023-21df00:30:122
24661272153,6sleep22466151-21apt-get00:25:142
24568152150,1sleep00-21swapper/000:16:390
23481122151,3sleep02348153-21kthreadcore23:10:160
22874902151,2sleep00-21swapper/022:32:280
22217102156,4sleep0151ktimers/021:51:580
21838422151,9sleep12184555-21cut21:30:191
21805302151,6sleep02150347-21diskmemload21:26:250
21804642151,9sleep22180475-21ssh21:26:182
20400362150,2sleep0171rcu_preempt19:25:170
202144399151,9cyclictest2144826-21taskset21:02:103
202144399151,9cyclictest2102671-21taskset20:23:093
202144399151,9cyclictest2070951-21taskset19:52:363
202144399150,12cyclictest0-21swapper/319:55:283
202144399150,12cyclictest0-21swapper/319:30:263
24852512141,3sleep02150347-21diskmemload00:35:230
24565062141,4sleep22456515-21ssh00:16:002
24245672141,7sleep02424620-21ntpq23:55:220
24092922140,9sleep22409307-21ssh23:45:372
23629522141,7sleep22150347-21diskmemload23:19:232
22885052141,6sleep22150347-21diskmemload22:34:022
22781772140,2sleep10-21swapper/122:26:101
22724312140,2sleep00-21swapper/022:24:580
22706712141,7sleep12150347-21diskmemload22:22:141
22543902140,2sleep20-21swapper/222:12:172
22452312141,8sleep12245241-21ssh22:05:351
22290632141,3sleep12229073-21ssh21:55:451
22215732141,6sleep20-21swapper/221:51:452
21731392140,2sleep20-21swapper/221:22:342
21007822141,8sleep22101683-21sed20:20:212
20719752141,8sleep02073129-21kthreadcore19:55:190
24732292131,8sleep12473230-21ssh00:29:431
24718402130,8sleep02471850-21ssh00:27:130
24511732131,7sleep2246-21systemd-journal00:15:012
24415912131,6sleep12150347-21diskmemload00:06:181
24404282130,1sleep00-21swapper/000:05:200
24195662130,2sleep02150347-21diskmemload23:54:090
23782602132,7sleep00-21swapper/023:27:560
23310522131,5sleep00-21swapper/023:00:100
23280162131,5sleep00-21swapper/022:56:080
22980392131,2sleep00-21swapper/022:40:130
22806032131,6sleep00-21swapper/022:29:540
22572472131,7sleep22258771-21cut22:15:182
22572472131,7sleep22258771-21cut22:15:182
22311852130,6sleep00-21swapper/021:59:040
22045592131,1sleep10-21swapper/121:40:561
21741932137,2sleep10-21swapper/121:24:131
21183642130,1sleep10-21swapper/120:36:001
20401352131,7sleep12041638-21kthreadcore19:25:181
24806012121,3sleep02480611-21ssh00:34:500
24742892121,7sleep12474732-21cat00:30:151
24589912125,3sleep20-21swapper/200:20:142
24293102123,5sleep12430917-21grep00:00:191
24042522121,5sleep00-21swapper/023:45:040
23933912120,1sleep00-21swapper/023:36:060
23687312121,5sleep10-21swapper/123:20:441
22458982121,5sleep20-21swapper/222:06:392
22377242121,1sleep20-21swapper/222:01:312
21905132121,4sleep02150347-21diskmemload21:34:220
21665462121,5sleep12166620-21munin-run21:20:011
21637362120,2sleep22150347-21diskmemload21:15:362
21582602121,5sleep00-21swapper/021:14:410
21574262121,3sleep22157436-21ssh21:13:242
21245112121,2sleep00-21swapper/020:45:160
20973592120,1sleep00-21swapper/020:18:140
20614302121,7sleep02062469-21cat19:45:170
24862762111,3sleep20-21swapper/200:36:242
24859142111,4sleep12150347-21diskmemload00:35:391
24653412111,4sleep12150347-21diskmemload00:24:101
24285622112,4sleep02428588-21apt-get00:00:140
24275292111,4sleep22150347-21diskmemload23:59:042
24142672111,6sleep22415986-21kthreadcore23:50:192
24013592111,3sleep20-21swapper/223:40:532
23992392111,5sleep12400280-21memory23:40:201
23959032111,6sleep22395932-21dump-pmu-power23:40:002
23939072111,4sleep12150347-21diskmemload23:36:551
23866592111,6sleep22150347-21diskmemload23:33:232
23860842111,4sleep12150347-21diskmemload23:32:301
23784542111,3sleep20-21swapper/223:28:132
23718302110,5sleep10-21swapper/123:25:121
23597482111,5sleep02359805-21ntpq23:15:210
23539482111,4sleep22150347-21diskmemload23:12:592
23451832111,2sleep20-21swapper/223:07:212
23391222111,3sleep10-21swapper/123:05:071
23371722111,4sleep22150347-21diskmemload23:02:292
23288052111,2sleep20-21swapper/222:57:262
23127172112,2sleep072-21kcompactd022:48:060
23122282111,4sleep22150347-21diskmemload22:47:202
23030702111,4sleep22150347-21diskmemload22:40:442
22946852111,4sleep12150347-21diskmemload22:35:571
22624832111,6sleep12150347-21diskmemload22:17:131
22624832111,6sleep12150347-21diskmemload22:17:121
22373832111,4sleep12150347-21diskmemload22:00:571
22309892111,4sleep22150347-21diskmemload21:58:452
22129732111,4sleep02150347-21diskmemload21:46:010
22059072111,3sleep20-21swapper/221:43:032
21886792111,3sleep20-21swapper/221:31:282
21123912111,5sleep02112440-21ntpq20:30:200
20877802111,2sleep20-21swapper/220:10:192
20701952111,5sleep22070249-21ntpq19:50:232
20560842110,7sleep12056224-21cut19:40:161
20382882111,5sleep12038334-21ntpq19:20:241
24411692100,2sleep20-21swapper/200:05:302
24019362100,3sleep00-21swapper/023:41:460
23849072101,3sleep02150347-21diskmemload23:30:400
22945992101,4sleep22150347-21diskmemload22:35:502
22894962101,4sleep02289520-21apt-get22:35:110
22051862101,3sleep00-21swapper/021:41:550
21968892101,3sleep00-21swapper/021:36:390
21806262101,4sleep12150347-21diskmemload21:26:341
21630012101,3sleep00-21swapper/021:15:220
21494012101,4sleep12149458-21ntp_states21:05:241
21432732101,5sleep02143890-21ls21:00:180
21299902101,5sleep02130071-21uname20:50:160
20819462101,4sleep12081965-21apt-get20:05:141
20807142101,5sleep22080762-21ntpq20:00:212
20456372101,4sleep22047333-21grep19:30:192
20256702101,4sleep02025716-21ntpq19:10:290
2144735290,3sleep12144716-21fwupd21:01:011
2144022291,3sleep22144068-21ntpq21:00:212
2135332291,4sleep22136635-21grep20:55:192
2135050291,4sleep02135259-21hddtemp_smartct20:55:180
2131582290,1sleep20-21swapper/220:50:192
2130074291,4sleep12131581-21cut20:50:181
2128476290,1sleep10-21swapper/120:45:251
2118979291,4sleep12119007-21grep20:40:151
2103079291,3sleep12103098-21apt-get20:25:131
2103079291,3sleep12103098-21apt-get20:25:121
2093372291,4sleep12095076-21grep20:15:201
2081683291,4sleep02081828-21cat20:04:590
2066131291,4sleep02066148-21/usr/sbin/munin19:50:140
2059985291,3sleep02060229-21users19:40:270
2056226293,3sleep244-21ksoftirqd/219:40:172
2124819281,3sleep22126387-21kthreadcore20:45:172
2104337283,2sleep22106076-21kthreadcore20:25:182
2104337283,2sleep22106076-21kthreadcore20:25:172
2102576280,2sleep172-21kcompactd020:22:001
2090318281,3sleep00-21swapper/020:10:220
2086634280,1sleep20-21swapper/220:05:592
2076388281,3sleep22076406-21latency_hist20:00:022
2073133281,3sleep10-21swapper/119:55:201
2054326281,3sleep02054363-21switchtime19:35:270
2048858282,3sleep12044781-21/usr/sbin/munin19:30:261
2043630280,4sleep22043721-21ps19:25:272
2035287281,3sleep20-21swapper/219:20:172
2021617281,3sleep12021632-21/usr/sbin/munin19:10:141
2113676270,2sleep02113695-21apt-get20:35:140
2097206270,1sleep20-21swapper/220:16:212
2070934270,1sleep10-21swapper/119:52:221
2065558270,1sleep10-21swapper/119:46:061
2064985271,3sleep22065075-21processes19:45:232
2050016271,1sleep10-21swapper/119:35:041
2049752270,1sleep00-21swapper/019:34:500
2033689270,1sleep00-21swapper/019:16:270
2033065271,3sleep10-21swapper/119:15:241
2032818270,2sleep20-21swapper/219:15:212
2150082260,1sleep20-21swapper/221:06:462
2145922262,1sleep0151ktimers/021:05:200
2123947260,3sleep00-21swapper/020:44:550
2118540260,1sleep20-21swapper/220:38:122
2113070260,4sleep20-21swapper/220:31:202
2106970260,2sleep00-21swapper/020:25:200
2106970260,2sleep00-21swapper/020:25:190
2097776260,4sleep00-21swapper/020:20:070
2082274261,2sleep02021434-21cyclictest20:05:170
2039188260,1sleep00-21swapper/019:23:590
2091340250,1sleep10-21swapper/120:10:261
2053922250,2sleep20-21swapper/219:35:212
2021633250,1sleep20-21swapper/219:10:152
2134923240,1sleep10-21swapper/120:55:151
2108651240,1sleep10-21swapper/120:30:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional