You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 04:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri May 01, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
332029999586,7cyclictest30-21ksoftirqd/220:35:032
332018999590,2cyclictest0-21swapper/019:50:020
332018999332,16cyclictest418357-21ssh00:15:020
33201899886,10cyclictest420400-21cron00:20:010
33201899864,12cyclictest334538-21modprobe20:00:000
33202999773,62cyclictest364455-21ssh22:10:022
332026997625,16cyclictest375077-21munin-run22:35:011
332033997332,37cyclictest361381-21sh22:03:143
332026997330,15cyclictest412048-21ssh00:00:081
332026997328,17cyclictest352499-21sshd21:42:391
332018997149,18cyclictest16850irq/134-800000023:05:030
33203399691,63cyclictest366129-21ssh22:14:193
33202999691,51cyclictest355570-21ssh21:50:012
332026996928,16cyclictest407334-21ssh23:49:411
332029996721,10cyclictest30-21ksoftirqd/222:16:442
332026996745,18cyclictest334283-21cron19:55:001
33203399662,60cyclictest382113-21ssh22:50:563
33203399661,61cyclictest398040-21ssh23:28:043
33203399661,61cyclictest388691-21ssh23:06:083
33203399661,61cyclictest381397-21ssh22:49:443
33203399661,60cyclictest48-21kcompactd021:55:233
332026996612,12cyclictest131rcu_preempt19:35:021
33203399652,59cyclictest391716-21ssh23:13:273
33203399651,60cyclictest417573-21ssh00:13:203
33203399651,60cyclictest371888-21ssh22:27:263
33203399651,60cyclictest367708-21ssh22:17:503
33203399651,59cyclictest353824-21ssh21:45:293
332033996426,34cyclictest335656-21latency_hist20:20:023
33203399642,59cyclictest423824-21ssh00:27:453
33203399642,58cyclictest414488-21ssh00:05:533
33203399641,59cyclictest420824-21ssh00:20:353
33203399641,59cyclictest411379-21ssh23:59:013
33203399641,59cyclictest408343-21ssh23:51:433
33203399641,59cyclictest406408-21ssh23:47:213
33203399641,59cyclictest404182-21ssh23:42:073
33203399641,59cyclictest404182-21ssh23:42:073
33203399641,59cyclictest378516-21ssh22:42:493
33203399641,59cyclictest374272-21ssh22:33:013
33203399641,59cyclictest362577-21ssh22:05:403
33203399641,59cyclictest356770-21ssh21:52:353
33203399641,59cyclictest345212-21ssh21:25:403
33203399641,59cyclictest342926-21ssh21:20:173
33203399641,59cyclictest338533-21ssh21:10:153
33203399641,58cyclictest332979-21sshd19:27:233
332026996426,15cyclictest379645-21ssh22:45:131
332026996426,13cyclictest150-21pr/ttyS221:48:191
33203399632,58cyclictest427449-21ssh00:35:553
33203399632,58cyclictest412230-21ssh00:00:343
33203399632,57cyclictest392808-21ssh23:15:393
33203399631,59cyclictest395852-21ssh23:22:563
33203399631,58cyclictest426012-21sshd00:32:473
33203399631,58cyclictest399491-21ssh23:31:083
33203399631,58cyclictest387272-21ssh23:02:593
33203399631,58cyclictest376537-21ssh22:38:263
33203399631,58cyclictest368990-21ssh22:20:283
33203399631,58cyclictest349646-21ssh21:35:593
33203399631,58cyclictest347379-21ssh21:30:433
33203399631,57cyclictest341340-21cron21:17:013
332026996336,21cyclictest388706-21ssh23:06:091
332026996325,8cyclictest369245-21ssh22:21:111
33203399621,58cyclictest418659-21ssh00:15:363
33203399621,58cyclictest352992-21ssh21:44:013
33203399621,58cyclictest335053-21sshd20:07:293
33203399621,57cyclictest401368-21ssh23:35:343
33203399621,56cyclictest384156-21ssh22:55:353
332029996218,9cyclictest30-21ksoftirqd/221:32:092
33203399611,56cyclictest334813-21sshd20:04:473
33203399611,55cyclictest333298-21cut19:35:023
332026996145,12cyclictest364766-21sh22:10:461
332018996124,31cyclictest338199-21diskmemload22:50:000
33203399601,55cyclictest336595-21sshd20:36:463
33203399601,55cyclictest336353-21sshd20:32:463
33203399601,54cyclictest48-21kcompactd020:44:483
332026996043,14cyclictest414335-21ssh00:05:321
332026996031,21cyclictest403104-21cron23:40:001
332026996023,14cyclictest423799-21ssh00:27:321
332026996023,13cyclictest345294-21ssh21:25:511
33201899602,3cyclictest131rcu_preempt22:55:000
33203399591,54cyclictest337124-21sshd20:45:463
33203399591,54cyclictest337124-21sshd20:45:463
33203399591,53cyclictest48-21kcompactd021:09:453
332029995915,11cyclictest30-21ksoftirqd/222:29:082
332026995944,12cyclictest427265-21ssh00:35:301
332026995944,12cyclictest393449-21ssh23:17:231
332026995941,14cyclictest351204-21cron21:39:591
332018995932,11cyclictest394149-21ssh23:19:110
332018995929,14cyclictest400521-21ssh23:33:580
33201899591,3cyclictest150-21pr/ttyS221:34:450
33203399581,52cyclictest335856-21sshd20:24:463
33203399581,51cyclictest48-21kcompactd020:13:093
33203399581,51cyclictest48-21kcompactd019:54:503
33203399581,51cyclictest333554-21munin-run19:40:013
332026995843,12cyclictest362542-21ssh22:05:341
332026995842,13cyclictest347355-21ssh21:30:381
33203399571,52cyclictest332767-21munin-run19:25:003
33203399571,51cyclictest48-21kcompactd021:00:523
33203399571,51cyclictest48-21kcompactd020:27:013
33203399571,50cyclictest337544-21cat20:55:013
332029995718,7cyclictest30-21ksoftirqd/220:45:012
332029995711,14cyclictest339803-21ssh21:13:412
332026995742,12cyclictest391157-21ssh23:11:571
332026995741,13cyclictest386651-21ssh23:01:221
332026995738,14cyclictest335320-21cron20:14:591
332018995747,6cyclictest16850irq/134-800000023:00:010
33201899573,17cyclictest407487-21grep23:49:590
33203399561,50cyclictest48-21kcompactd019:58:103
33203399561,50cyclictest48-21kcompactd019:11:353
332029995614,8cyclictest363160-21ssh22:07:132
332026995643,10cyclictest343119-21ssh21:20:551
332026995641,12cyclictest385191-21ssh22:58:211
332026995620,32cyclictest368782-21cat22:20:031
332026995614,13cyclictest396001-21ssh23:23:161
332018995642,8cyclictest16850irq/134-800000021:30:010
332018995628,12cyclictest401547-21ssh23:35:590
332018995627,13cyclictest349331-21ssh21:35:020
332018995625,15cyclictest344212-21ssh21:23:380
33203399551,49cyclictest364-21snmpd19:41:143
33203399551,49cyclictest364-21snmpd19:41:143
33203399551,49cyclictest364-21snmpd19:18:043
332026995542,10cyclictest339928-21ssh21:13:581
332026995540,12cyclictest410598-21ssh23:57:021
332026995537,14cyclictest371579-21ssh22:26:401
332026995532,19cyclictest399282-21ssh23:30:371
332026995532,18cyclictest416973-21ssh00:11:471
332026995530,21cyclictest341881-21sh21:18:191
332026995528,23cyclictest360457-21rm22:00:531
33201899555,3cyclictest131rcu_preempt20:24:460
33201899554,41cyclictest333501-21cron19:40:000
332018995542,10cyclictest275-21dbus-daemon22:35:000
332018995534,16cyclictest405287-21ssh23:45:010
332018995534,16cyclictest405287-21ssh23:45:000
332018995529,10cyclictest395406-21ssh23:21:400
332018995527,13cyclictest360926-21ssh22:01:580
33201899552,3cyclictest131rcu_preempt23:59:460
332026995442,9cyclictest358061-21ssh21:55:231
332026995441,11cyclictest421687-21ssh00:22:461
332026995437,14cyclictest408152-21ssh23:51:151
332026995437,13cyclictest425134-21ssh00:30:331
332026995436,14cyclictest381964-21ssh22:50:251
332026995433,17cyclictest419604-21ssh00:17:561
332026995428,21cyclictest332143-21latency_hist19:10:021
332018995428,10cyclictest367058-21ssh22:16:100
332018995425,13cyclictest351632-21ssh21:40:260
332018995424,14cyclictest390546-21ssh23:10:170
332018995423,16cyclictest414623-21ssh00:06:130
332018995420,16cyclictest421990-21ssh00:23:270
332018995418,32cyclictest12-21ksoftirqd/021:14:000
332029995321,10cyclictest30-21ksoftirqd/222:59:402
332026995341,9cyclictest405146-21ssh23:44:351
332026995341,9cyclictest405146-21ssh23:44:341
332026995341,9cyclictest334511-21sshd19:55:451
332026995341,8cyclictest48-21kcompactd020:37:011
332026995339,11cyclictest379164-21ssh22:44:301
332026995339,11cyclictest356204-21ssh21:51:051
332026995338,11cyclictest336125-21munin-run20:30:001
332026995337,13cyclictest375780-21ssh22:36:241
33202699533,35cyclictest221rcuc/121:10:021
332026995333,16cyclictest397234-21ssh23:25:511
332026995333,15cyclictest332482-21munin-plugin-st19:20:011
332026995327,21cyclictest338069-21latency_hist21:05:021
332018995329,8cyclictest12-21ksoftirqd/021:55:000
33201899531,37cyclictest379169-21ssh22:44:290
332026995239,9cyclictest1-21systemd19:25:001
332026995233,15cyclictest333760-21sshd19:41:461
332026995233,15cyclictest333760-21sshd19:41:461
332026995229,18cyclictest336340-21sshd20:30:461
332018995226,11cyclictest424060-21ssh00:28:190
332018995225,12cyclictest407824-21ssh23:50:270
33203399511,46cyclictest334016-21sshd19:47:273
33202999512,5cyclictest131rcu_preempt22:49:492
332026995139,9cyclictest48-21kcompactd020:20:551
332018995144,4cyclictest16850irq/134-800000000:35:020
33201899514,3cyclictest131rcu_preempt21:57:530
332018995129,16cyclictest275-21dbus-daemon23:05:000
332029995041,5cyclictest337966-21kworker/2:200:35:152
332029995014,6cyclictest30-21ksoftirqd/223:30:272
332018995023,12cyclictest341366-21ssh21:17:060
33203399491,44cyclictest3467562-21sshd20:59:463
33203399491,44cyclictest332117-21sed19:10:013
332029994931,12cyclictest309-21in:imuxsock19:35:022
332029994915,11cyclictest30-21ksoftirqd/223:35:562
332026994935,10cyclictest364-21snmpd19:46:171
332026994931,15cyclictest333664-21latency_hist19:40:021
332026994930,15cyclictest337811-21cat21:00:021
332026994927,18cyclictest337063-21latency_hist20:45:021
332026994926,18cyclictest337298-21latency_hist20:50:021
332026994926,18cyclictest337298-21latency_hist20:50:021
332026994926,18cyclictest335822-21latency_hist20:20:021
33201899494,38cyclictest336616-21cron20:40:000
33201899492,40cyclictest366480-21latency_hist22:15:030
332018994922,12cyclictest412928-21ssh00:02:250
33202999489,9cyclictest0-21swapper/200:18:152
332029994840,5cyclictest337966-21kworker/2:221:55:022
332029994840,5cyclictest336683-21kworker/2:123:54:122
332029994832,12cyclictest1-21systemd20:15:012
332029994813,7cyclictest30-21ksoftirqd/221:28:422
332026994832,12cyclictest332377-21cat19:15:021
332026994827,14cyclictest334822-21modprobe20:05:001
332018994839,5cyclictest16850irq/134-800000000:35:370
33202999474,14cyclictest392666-21ssh23:15:192
332029994734,10cyclictest344749-21systemd-run21:25:002
332029994732,12cyclictest1-21systemd19:40:002
332029994711,5cyclictest0-21swapper/222:40:572
332026994733,10cyclictest335067-21cron20:10:001
332026994729,14cyclictest364-21snmpd20:50:221
332018994735,9cyclictest12-21ksoftirqd/019:50:000
332018994735,8cyclictest12-21ksoftirqd/020:35:000
332018994734,7cyclictest377169-21modprobe22:40:010
33201899472,39cyclictest398801-21sh23:29:590
332018994721,11cyclictest370733-21ssh22:25:000
33201899471,32cyclictest151rcuc/020:45:030
33201899471,32cyclictest151rcuc/020:45:020
332018994712,9cyclictest12-21ksoftirqd/021:08:460
332026994631,10cyclictest364-21snmpd19:28:231
33201899465,9cyclictest275-21dbus-daemon19:20:010
33201899464,6cyclictest372869-21munin-run22:30:000
33201899463,38cyclictest275-21dbus-daemon22:10:010
33201899463,37cyclictest0-21swapper/020:20:010
33202999451,8cyclictest396685-21unin-run23:25:012
332029994513,6cyclictest30-21ksoftirqd/223:02:442
332018994529,12cyclictest12-21ksoftirqd/019:25:010
332018994515,14cyclictest353912-21ssh21:45:500
332029994429,10cyclictest3467602-21systemd-journal23:10:012
33202999439,5cyclictest131rcu_preempt22:31:542
332029994335,5cyclictest337966-21kworker/2:200:33:562
332029994322,17cyclictest398868-21sed23:30:012
332029994312,6cyclictest30-21ksoftirqd/223:48:152
332018994336,4cyclictest16950irq/224-800000019:15:010
332029994232,6cyclictest337966-21kworker/2:221:40:012
332029994227,12cyclictest414360-21ssh00:05:352
332029994223,14cyclictest30-21ksoftirqd/200:00:152
332029994212,5cyclictest30-21ksoftirqd/222:38:322
332029994210,8cyclictest337966-21kworker/2:221:20:002
33201899428,18cyclictest337661-21modprobe21:00:000
332018994226,12cyclictest12-21ksoftirqd/019:34:410
33201899422,15cyclictest333807-21unin-run19:45:010
33201899422,15cyclictest333807-21unin-run19:45:010
33202999414,6cyclictest30-21ksoftirqd/222:00:322
332029994134,4cyclictest337966-21kworker/2:200:11:002
332029994131,7cyclictest30-21ksoftirqd/222:50:182
332029994127,11cyclictest30-21ksoftirqd/200:25:022
332029994125,12cyclictest422851-21ssh00:25:092
332029994112,6cyclictest131rcu_preempt19:20:002
332018994134,4cyclictest16950irq/224-800000020:10:010
332018994131,6cyclictest19350irq/34-mmc119:27:560
33202999404,5cyclictest30-21ksoftirqd/221:55:162
332029994028,8cyclictest353584-21date21:45:032
332029994026,9cyclictest3467602-21systemd-journal20:55:002
332029994026,11cyclictest392448-21sed23:15:012
332029994025,12cyclictest30-21ksoftirqd/223:56:422
332029994022,13cyclictest334284-21cron19:55:002
332029994016,19cyclictest369235-21ssh22:21:112
33201899404,6cyclictest334830-21munin-run20:05:000
332018994033,4cyclictest16950irq/224-800000020:15:010
332018994026,11cyclictest12-21ksoftirqd/020:45:010
332029993926,10cyclictest335582-21cron20:20:002
332029993922,12cyclictest1-21systemd20:30:002
332029993922,11cyclictest3467602-21systemd-journal21:10:002
332029993823,11cyclictest275-21dbus-daemon19:30:012
33202999382,10cyclictest30-21ksoftirqd/220:55:462
33201899382,32cyclictest0-21swapper/020:30:020
33202999376,7cyclictest30-21ksoftirqd/219:11:112
33202999373,7cyclictest30-21ksoftirqd/223:40:132
33202999373,7cyclictest30-21ksoftirqd/223:40:132
332029993725,9cyclictest30-21ksoftirqd/221:05:012
332029993719,14cyclictest275-21dbus-daemon19:25:012
33201899371,32cyclictest0-21swapper/020:50:500
332029993625,9cyclictest30-21ksoftirqd/220:34:462
332029993519,11cyclictest30-21ksoftirqd/220:03:462
332029993516,15cyclictest48-21kcompactd020:05:232
332029993510,19cyclictest275-21dbus-daemon19:45:012
332029993510,19cyclictest275-21dbus-daemon19:45:012
332018993516,15cyclictest337926-21sshd21:03:460
33202999344,6cyclictest30-21ksoftirqd/219:49:462
332029993426,5cyclictest332028-21latency_hist19:10:022
332029993414,15cyclictest335980-21sed20:25:022
332018993426,6cyclictest12-21ksoftirqd/019:10:020
332029993218,11cyclictest337160-21munin-plugin-st20:50:002
332029993218,11cyclictest337160-21munin-plugin-st20:50:002
33202999319,19cyclictest334525-21sshd19:58:472
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional