You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 10:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed May 06, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1934096999427,13cyclictest1961043-21ssh21:52:580
193409699931,37cyclictest1987256-21ssh22:47:130
193409699918,10cyclictest2005264-21modprobe23:25:010
1934096998529,14cyclictest2028188-21ssh00:12:120
1934096998527,15cyclictest1983057-21ssh22:38:420
1934096998427,12cyclictest1974797-21ssh22:21:240
193409699842,41cyclictest1951452-21ssh21:33:060
1934096998323,25cyclictest1954354-21ssh21:39:080
1934096998119,5cyclictest16850irq/134-800000021:55:520
1934096998014,6cyclictest16850irq/134-800000022:10:300
1934096997919,14cyclictest2019685-21ssh23:54:530
193410399780,61cyclictest0-21swapper/220:50:032
1934096997828,17cyclictest1989178-21ssh22:51:040
1934096997525,4cyclictest0-21swapper/023:35:160
1934096997518,48cyclictest16850irq/134-800000000:37:220
1934096997429,10cyclictest2008620-21ssh23:31:350
1934096997423,13cyclictest2002298-21ssh23:18:420
1934096997417,50cyclictest16850irq/134-800000000:33:010
1934096997414,56cyclictest12-21ksoftirqd/022:17:490
1934096997414,18cyclictest16850irq/134-800000022:59:500
1934096997410,6cyclictest16850irq/134-800000022:32:310
193409699737,17cyclictest16850irq/134-800000022:07:100
193409699733,6cyclictest2007733-21munin-run23:30:000
1934096997326,14cyclictest2035201-21ssh00:26:440
1934096997326,14cyclictest2035201-21ssh00:26:430
1934096997326,13cyclictest1948549-21ssh21:26:480
1934096997324,14cyclictest1985124-21ssh22:42:560
1934096997324,14cyclictest1985124-21ssh22:42:550
1934096997316,52cyclictest16850irq/134-800000023:12:350
1934103997230,14cyclictest1940269-21diskmemload23:41:142
193409699722,9cyclictest2022207-21systemd-run00:00:020
193409699722,9cyclictest2022207-21systemd-run00:00:020
1934096997217,50cyclictest16850irq/134-800000021:47:470
1934096997217,45cyclictest16850irq/134-800000021:24:060
1934106997124,40cyclictest1-21systemd00:35:013
1934096997126,12cyclictest1994724-21ssh23:02:510
1934096997126,10cyclictest2025367-21ssh00:06:170
1934096997126,10cyclictest2025367-21ssh00:06:160
1934096997125,13cyclictest1944833-21ssh21:19:220
1934096997121,12cyclictest1965647-21ssh22:02:230
1934096997120,14cyclictest2015818-21ssh23:46:320
1934106997028,38cyclictest1939903-21cat21:00:013
1934096997022,8cyclictest2030031-21ssh00:15:580
1934096997013,47cyclictest16850irq/134-800000000:23:170
1934101996928,37cyclictest1951249-21sh21:32:321
1934096996930,34cyclictest19350irq/34-mmc119:55:020
193409699691,3cyclictest2014948-21modprobe23:45:000
1934101996819,44cyclictest3467602-21systemd-journal22:05:011
1934096996823,12cyclictest1976763-21ssh22:25:180
1934096996823,12cyclictest1976763-21ssh22:25:180
1934096996822,15cyclictest1996877-21ssh23:07:220
193409699681,39cyclictest2023630-21ssh00:02:460
1934103996753,11cyclictest2032891-21ssh00:21:532
1934103996722,14cyclictest1940269-21diskmemload21:52:262
1934106996624,38cyclictest37-21ksoftirqd/323:46:453
1934103996648,15cyclictest1980270-21ssh22:32:582
1934103996648,14cyclictest1966698-21ssh22:04:532
1934101996624,38cyclictest2021076-21sh23:57:281
1934101996624,38cyclictest2021076-21sh23:57:281
1934096996619,10cyclictest1934135-21wc19:10:010
1934103996552,10cyclictest1971925-21ssh22:15:252
193410199653,14cyclictest2002822-21ssh23:19:591
193409699652,21cyclictest1957158-21unin-run21:45:010
1934096996522,15cyclictest275-21dbus-daemon19:35:000
193410699641,16cyclictest1990269-21ssh22:53:403
1934103996452,9cyclictest2008109-21ssh23:30:242
1934103996448,13cyclictest1967101-21ssh22:05:162
1934103996448,13cyclictest1940600-21ssh21:10:152
1934103996447,13cyclictest2016397-21ssh23:47:522
1934103996447,13cyclictest2011491-21ssh23:37:472
1934103996447,13cyclictest1977908-21ssh22:28:012
1934103996447,13cyclictest1977908-21ssh22:28:002
1934103996447,13cyclictest1970155-21ssh22:11:542
1934103996447,13cyclictest1962911-21ssh21:56:432
1934103996446,14cyclictest2028115-21ssh00:11:582
1934103996446,14cyclictest1988142-21ssh22:49:132
1934103996446,14cyclictest1983975-21ssh22:40:152
1934103996446,14cyclictest1983975-21ssh22:40:152
193410199643,16cyclictest2012534-21ssh23:40:011
193410699634,9cyclictest1935318-21cron19:35:003
1934106996339,20cyclictest37-21ksoftirqd/323:45:013
193410699632,12cyclictest1983611-21cron22:40:013
1934103996350,11cyclictest1948937-21ssh21:27:452
1934103996348,12cyclictest2030485-21ssh00:16:582
1934103996348,12cyclictest1999199-21ssh23:12:072
1934103996348,12cyclictest1991550-21ssh22:56:022
1934103996347,13cyclictest2023402-21ssh00:02:162
1934103996347,13cyclictest1990254-21ssh22:53:392
1934103996347,12cyclictest2040868-21ssh00:38:412
1934103996346,14cyclictest2004188-21ssh23:22:312
1934103996346,14cyclictest1997630-21ssh23:09:062
1934103996346,14cyclictest1955555-21ssh21:41:192
1934103996346,13cyclictest2037889-21ssh00:32:212
1934103996346,13cyclictest2026939-21ssh00:09:562
1934103996346,13cyclictest2026939-21ssh00:09:552
1934103996346,13cyclictest2002634-21ssh23:19:262
1934103996346,13cyclictest1983472-21ssh22:39:432
1934103996346,13cyclictest1974822-21ssh22:21:272
1934103996346,13cyclictest1953390-21ssh21:37:002
1934103996346,13cyclictest1946263-21ssh21:22:012
1934103996345,14cyclictest2018972-21ssh23:53:152
193410199632,15cyclictest1936603-21cron20:00:001
1934096996334,26cyclictest1936892-21latency_hist20:04:590
1934096996328,30cyclictest16950irq/224-800000021:11:160
1934106996223,34cyclictest1935614-21modprobe19:40:003
193410699622,13cyclictest364-21snmpd22:10:113
1934106996221,36cyclictest310-21in:imklog19:49:173
1934106996220,38cyclictest1993272-21latency_hist23:00:033
1934106996218,39cyclictest309-21in:imuxsock21:50:013
1934106996218,37cyclictest2010183-21cut23:35:013
1934103996249,10cyclictest2036515-21ssh00:29:472
1934103996249,10cyclictest2036515-21ssh00:29:462
1934103996246,13cyclictest1994083-21ssh23:01:192
1934103996246,13cyclictest1958916-21ssh21:48:342
1934103996246,13cyclictest1951966-21ssh21:34:092
1934103996246,13cyclictest1944039-21ssh21:17:302
1934103996246,12cyclictest2020447-21ssh23:56:012
1934103996246,12cyclictest2020447-21ssh23:56:002
1934103996245,14cyclictest2006514-21ssh23:27:242
1934101996230,29cyclictest2017348-21latency_hist23:50:021
1934101996224,33cyclictest1981314-21cat22:35:011
1934101996220,38cyclictest3467602-21systemd-journal20:59:011
1934101996220,34cyclictest0-21swapper/121:40:011
1934101996219,4cyclictest1962404-21sh21:55:311
1934101996218,9cyclictest1970069-21sh22:11:441
1934101996216,11cyclictest1987333-1kworker/u9:122:48:591
193409699624,19cyclictest1934532-21cron19:20:000
1934106996125,32cyclictest1-21systemd20:10:013
1934106996121,35cyclictest1-21systemd23:55:013
1934106996119,37cyclictest37-21ksoftirqd/323:40:003
1934106996114,8cyclictest1950119-21latency_hist21:30:023
193410199613,16cyclictest2026999-21munin-run00:10:011
193410199613,16cyclictest2026999-21munin-run00:10:001
1934101996120,37cyclictest1957157-21systemd-run21:45:011
1934101996118,4cyclictest2029071-21sh00:14:171
193409699614,15cyclictest1935178-21grep19:30:030
193409699613,20cyclictest1-21systemd20:55:020
193409699613,20cyclictest1-21systemd20:55:010
193410699602,18cyclictest1960393-21ssh21:51:283
193410699602,14cyclictest1967160-21ssh22:05:233
193410699602,10cyclictest2022137-40mandb00:00:043
193410699602,10cyclictest2022137-40mandb00:00:033
1934106996011,4cyclictest0-21swapper/322:24:143
1934103996044,12cyclictest1937381-21sshd20:14:382
1934103996044,12cyclictest1937381-21sshd20:14:372
1934101996027,29cyclictest2013719-21sh23:42:151
1934101996025,31cyclictest1942389-21sh21:14:141
1934101996023,32cyclictest23-21ksoftirqd/100:27:501
1934101996023,32cyclictest23-21ksoftirqd/100:27:501
193410199602,16cyclictest3467602-21systemd-journal21:05:371
1934101996020,5cyclictest1946085-21sshd21:21:401
1934101996020,36cyclictest2017877-21sh23:50:451
1934101996016,9cyclictest1980481-1kworker/u9:122:36:381
1934106995928,25cyclictest1954710-21grep21:40:003
193410699592,18cyclictest1957099-21ssh21:44:513
193410699592,14cyclictest2000839-21ssh23:15:233
193410699591,16cyclictest1946972-21sshd21:23:383
1934106995910,12cyclictest1-21systemd23:30:013
193410199593,11cyclictest275-21dbus-daemon21:30:011
1934101995923,31cyclictest275-21dbus-daemon23:15:011
1934101995922,32cyclictest3467562-21sshd22:20:371
193410199592,19cyclictest364-21snmpd00:21:531
193410199592,17cyclictest2023575-21ssh00:02:391
193410199592,11cyclictest1935570-21cron19:40:001
1934101995919,4cyclictest2029830-21sh00:15:241
193410199591,14cyclictest1993686-21ssh23:00:291
1934096995930,26cyclictest16950irq/224-800000020:09:380
193410699583,15cyclictest1937828-21wc20:20:023
193410699582,17cyclictest1993819-21ssh23:00:453
193410699582,14cyclictest275-21dbus-daemon19:15:013
1934106995821,32cyclictest1940269-21diskmemload22:42:243
1934106995821,32cyclictest1940269-21diskmemload22:42:233
193410699582,10cyclictest1940253-21cron21:10:003
1934106995818,4cyclictest1940269-21diskmemload00:11:153
1934103995846,8cyclictest48-21kcompactd019:26:252
193410199582,18cyclictest1960353-21ssh21:51:231
193409699585,15cyclictest1939734-21modprobe21:00:000
1934106995722,31cyclictest1965878-21sh22:02:593
1934106995722,30cyclictest1963164-21sh21:57:153
193410699572,14cyclictest1939618-21grep20:55:033
193410699572,14cyclictest1939618-21grep20:55:023
193410699572,14cyclictest1936665-21cat20:00:013
193410699572,13cyclictest2003860-21ssh23:21:513
1934106995718,33cyclictest1939988-21kworker/u8:121:20:013
1934106995717,7cyclictest1981186-21cron22:34:593
193410699571,18cyclictest2041403-21cron00:40:013
1934103995745,9cyclictest48-21kcompactd020:30:462
193410199573,16cyclictest1996596-21ssh23:06:371
193410199573,12cyclictest2007845-21cat23:30:021
1934101995718,7cyclictest23-21ksoftirqd/121:48:331
1934101995713,6cyclictest23-21ksoftirqd/122:17:451
193410199571,15cyclictest1984489-21ssh22:41:271
193410199571,15cyclictest1984489-21ssh22:41:271
193409699577,19cyclictest1936628-21systemctl20:00:000
193409699575,21cyclictest1938705-21systemctl20:40:010
193409699575,21cyclictest1938705-21systemctl20:40:000
193410699563,15cyclictest2027165-21cut00:10:033
193410699563,15cyclictest2027165-21cut00:10:023
1934106995626,26cyclictest37-21ksoftirqd/322:17:043
1934106995625,26cyclictest2034681-21sh00:25:313
1934106995625,26cyclictest2034681-21sh00:25:303
1934106995618,34cyclictest1938082-21sort20:25:023
1934106995618,34cyclictest1936546-21cat19:55:033
1934106995615,4cyclictest1986781-21sh22:46:073
193410699561,15cyclictest2030723-21ssh00:17:333
193410699561,15cyclictest1935873-21pluginstate19:45:023
193410699561,15cyclictest1935873-21pluginstate19:45:013
193410699561,14cyclictest1999583-21ssh23:13:083
1934103995644,8cyclictest48-21kcompactd020:38:332
1934103995644,8cyclictest48-21kcompactd020:38:332
193410199563,11cyclictest1938685-21modprobe20:40:011
193410199563,11cyclictest1938685-21modprobe20:40:001
1934101995622,31cyclictest23-21ksoftirqd/122:58:341
1934101995622,30cyclictest2040851-21sh00:38:401
193409699565,15cyclictest1940261-21modprobe21:10:000
1934106995531,20cyclictest1934824-21munin-plugin-st19:25:003
193410699552,14cyclictest1998041-21ssh23:09:583
1934103995543,8cyclictest364-21snmpd19:48:092
193410199556,43cyclictest0-21swapper/121:20:001
1934101995523,28cyclictest2009614-21sh23:33:491
193410199552,18cyclictest1934534-21cron19:20:001
193410199552,17cyclictest3467602-21systemd-journal20:55:011
193410199552,17cyclictest3467602-21systemd-journal20:55:001
193410199552,15cyclictest1967170-21ssh22:05:241
193410199552,14cyclictest1-21systemd20:05:001
1934101995513,4cyclictest0-21swapper/120:11:331
1934101995513,4cyclictest0-21swapper/120:11:321
193409699555,17cyclictest1935579-21modprobe19:40:000
193409699555,14cyclictest364-21snmpd20:25:000
193409699554,15cyclictest3467602-21systemd-journal19:15:010
1934106995419,32cyclictest1940269-21diskmemload00:04:103
1934106995419,30cyclictest1934565-21tr19:20:013
1934106995411,4cyclictest0-21swapper/320:03:113
1934103995442,9cyclictest48-21kcompactd019:10:202
193410199546,11cyclictest1934278-21modprobe19:15:001
193410199544,16cyclictest1937209-21latency_hist20:10:011
1934101995427,23cyclictest23-21ksoftirqd/120:20:011
193410199542,17cyclictest2037596-21ssh00:31:441
193410199542,16cyclictest1939241-21munin-run20:50:011
1934101995414,4cyclictest1989664-21sh22:52:131
193410699533,11cyclictest1-21systemd20:40:023
193410699533,11cyclictest1-21systemd20:40:013
1934106995329,21cyclictest37-21ksoftirqd/321:34:383
193410699532,18cyclictest2032272-21ssh00:20:263
193410699532,14cyclictest1939193-21sshd20:46:363
1934106995321,28cyclictest1977761-21sh22:27:423
1934106995321,28cyclictest1977761-21sh22:27:413
193410699531,13cyclictest1941304-21ssh21:11:483
1934103995341,9cyclictest48-21kcompactd020:59:382
1934103995340,10cyclictest364-21snmpd21:01:382
193410199533,13cyclictest2004991-21ssh23:24:191
193410199532,15cyclictest1936336-21modprobe19:51:371
1934101995313,7cyclictest1935212-21latency_hist19:30:021
193409699533,21cyclictest1935867-21true19:45:010
193409699533,21cyclictest1935867-21true19:45:000
193410699522,14cyclictest1937398-21munin-run20:15:003
193410699522,14cyclictest1937398-21munin-run20:15:003
1934106995216,7cyclictest37-21ksoftirqd/319:28:253
1934103995245,4cyclictest1935844-21sshd19:44:372
1934103995245,4cyclictest1935844-21sshd19:44:372
193410199523,11cyclictest1940072-21grep21:05:011
1934103995145,3cyclictest1934507-21sshd19:16:372
1934103995142,5cyclictest48-21kcompactd019:30:512
193410699502,15cyclictest1939115-21wc20:45:023
193410699502,12cyclictest1940012-21munin-run21:05:013
1934106995011,6cyclictest310-21in:imklog20:31:373
1934103995043,3cyclictest48-21kcompactd020:54:382
1934103995043,3cyclictest48-21kcompactd020:54:382
1934103995038,9cyclictest3467562-21sshd19:22:372
1934103995033,13cyclictest3467602-21systemd-journal21:05:022
1934103995033,13cyclictest1938980-21/usr/sbin/munin20:45:002
1934103995030,16cyclictest1935546-21sshd19:35:382
1934101995032,13cyclictest3467602-21systemd-journal19:35:001
193410199502,10cyclictest1934213-21latency_hist19:10:021
1934103994942,3cyclictest48-21kcompactd020:04:422
193410199492,16cyclictest1936126-21munin-run19:50:001
193410199491,15cyclictest3467602-21systemd-journal20:43:411
193409699495,13cyclictest309-21in:imuxsock20:20:000
1934103994836,9cyclictest48-21kcompactd020:26:542
193410199483,14cyclictest1934826-21munin-run19:25:011
193410199482,15cyclictest1977938-21ssh22:28:051
193410199482,15cyclictest1977938-21ssh22:28:051
193410199481,17cyclictest1938193-21munin-run20:30:001
1934101994715,5cyclictest1938438-21sh20:35:011
193410699462,14cyclictest1938206-21munin-run20:30:013
1934103994633,9cyclictest364-21snmpd19:59:432
193410199463,15cyclictest1935861-21munin-run19:45:011
193410199463,15cyclictest1935861-21munin-run19:45:001
193410199462,12cyclictest1937938-21modprobe20:25:011
193409699461,41cyclictest1940024-21systemd-run21:05:010
1934103994535,7cyclictest0-21swapper/219:51:562
1934103994534,8cyclictest0-21swapper/220:16:512
1934096994432,9cyclictest3467602-21systemd-journal19:45:370
1934096994323,4cyclictest16850irq/134-800000020:15:020
1934096994323,4cyclictest16850irq/134-800000020:15:010
1934103994232,7cyclictest3467562-21sshd20:05:362
1934096994226,12cyclictest1938946-21sshd20:41:370
1934096994134,4cyclictest16950irq/224-800000020:35:000
1934103993930,7cyclictest0-21swapper/220:20:332
1934096993932,4cyclictest16950irq/224-800000019:22:370
1934103993826,9cyclictest0-21swapper/219:10:022
1934096993720,14cyclictest275-21dbus-daemon20:50:010
193409699366,24cyclictest1938194-21grep20:30:010
1934106992718,7cyclictest1934200-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional