You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 21:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Aug 04, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
194707999110102,3cyclictest0-21swapper/010:10:030
194708899891,3cyclictest131rcu_preempt12:15:022
194708899891,3cyclictest131rcu_preempt12:15:022
1947084998731,25cyclictest1968923-21sh09:42:451
1947084998724,33cyclictest2048294-21latency_hist12:30:021
194708499871,48cyclictest1983578-21sh10:13:421
194708499852,54cyclictest1994795-21sh10:37:361
1947084998520,34cyclictest1950335-21sed08:10:031
194708499845,6cyclictest131rcu_preempt08:50:031
194708499845,6cyclictest131rcu_preempt08:50:021
194708499841,50cyclictest1978002-21ssh10:01:411
194708499841,46cyclictest2017230-21sh11:24:501
194708499821,47cyclictest2048755-21ssh12:30:381
194708499791,51cyclictest1954245-21ssh09:12:021
194708499781,2cyclictest0-21swapper/108:50:021
194708499771,45cyclictest2025592-21ssh11:42:021
194708499771,45cyclictest2025592-21ssh11:42:011
194708499762,38cyclictest2033739-21ssh11:59:251
1947084997612,34cyclictest1953256-21diskmemload11:18:501
194708499741,44cyclictest1981094-21ssh10:08:271
194707999749,38cyclictest0-21swapper/010:37:380
1947084997328,40cyclictest275-21dbus-daemon12:40:001
194708499731,44cyclictest2001319-21rm10:51:161
194708499731,44cyclictest2001319-21rm10:51:151
194708499731,42cyclictest2040057-21ssh12:12:271
194708499731,42cyclictest2040057-21ssh12:12:271
1947079997230,21cyclictest1988311-21ssh10:24:070
1947079997230,21cyclictest1988311-21ssh10:24:070
194708499712,43cyclictest1970307-21ssh09:45:331
194708499711,46cyclictest2023163-21ssh11:37:011
194708499701,46cyclictest2004999-21ssh10:59:161
1947079997033,34cyclictest16850irq/134-800000010:10:030
194708499692,42cyclictest2019750-21munin-run11:30:011
194708499692,38cyclictest1959032-21ssh09:21:591
1947079996827,20cyclictest1985433-21ssh10:17:220
194707999661,36cyclictest131rcu_preempt12:15:360
194708499652,59cyclictest1949890-21munin-run08:05:001
1947079996551,8cyclictest16950irq/224-800000009:10:000
1947079996522,37cyclictest275-21dbus-daemon12:00:010
194708499642,33cyclictest1953256-21diskmemload09:54:551
194708499641,59cyclictest2028602-21ssh11:48:331
194708499641,59cyclictest2009334-21ssh11:08:101
194708499632,57cyclictest2011094-21ssh11:11:411
194708499631,58cyclictest2030478-21ssh11:52:291
194708499631,58cyclictest2007083-21ssh11:03:311
194708499631,58cyclictest2007083-21ssh11:03:301
194708499631,58cyclictest1984973-21ssh10:16:251
194708499631,58cyclictest1976110-21ssh09:57:581
194708499631,58cyclictest1963000-21ssh09:30:101
194708499631,58cyclictest1963000-21ssh09:30:091
194707999633,31cyclictest19350irq/34-mmc109:05:020
194707999632,4cyclictest131rcu_preempt11:40:000
1947093996213,6cyclictest131rcu_preempt12:40:013
1947084996228,31cyclictest23-21ksoftirqd/109:26:591
1947084996227,31cyclictest2021073-21sh11:32:411
194708499621,9cyclictest2042993-21ssh12:18:471
194708499621,8cyclictest1988062-21ssh10:23:361
194708499621,8cyclictest1988062-21ssh10:23:361
194708499621,57cyclictest2034770-21ssh12:01:111
194708499621,57cyclictest1967172-21ssh09:39:201
1947088996137,9cyclictest1972273-21kworker/2:010:09:252
194708499612,55cyclictest1989617-21ssh10:26:401
194708499612,53cyclictest1951148-21sshd08:25:541
194708499611,56cyclictest1999131-21ssh10:46:401
194708499611,56cyclictest1996874-21ssh10:41:581
194708499611,56cyclictest1992340-21ssh10:32:301
194708499611,56cyclictest1951450-21munin-run08:35:011
194708499611,55cyclictest1953230-21sshd09:08:541
194707999614,3cyclictest131rcu_preempt10:57:590
194707999613,3cyclictest131rcu_preempt09:59:350
1947079996128,15cyclictest2037261-21ssh12:06:360
1947093996011,6cyclictest131rcu_preempt11:20:423
194708499602,54cyclictest2036887-21ssh12:05:371
194708499601,6cyclictest1955860-21ssh09:15:191
194708499601,55cyclictest2044299-21ssh12:21:171
194708499601,55cyclictest1948082-21munin-run07:30:011
1947093995936,17cyclictest275-21dbus-daemon10:00:013
1947088995950,5cyclictest1986647-21kworker/2:310:38:302
194708499592,52cyclictest1951735-21latency_hist08:40:011
194708499592,52cyclictest1951735-21latency_hist08:40:011
1947079995948,7cyclictest16950irq/224-800000012:30:000
1947079995929,15cyclictest2048701-21ssh12:30:320
1947088995832,8cyclictest1981794-21kworker/2:212:25:092
1947088995830,9cyclictest1981794-21kworker/2:212:31:492
194708499582,51cyclictest364-21snmpd07:33:231
194708499582,51cyclictest1952990-21cron09:05:001
194707999581,3cyclictest131rcu_preempt11:29:010
194707999581,3cyclictest131rcu_preempt11:02:460
194707999581,3cyclictest131rcu_preempt11:02:450
194708499572,50cyclictest1952729-21munin-run09:00:001
194708499571,50cyclictest1947513-21sshd07:18:541
194707999571,3cyclictest131rcu_preempt11:13:210
194707999571,38cyclictest131rcu_preempt10:00:330
1947079995710,41cyclictest1965584-21kworker/u8:109:40:000
1947088995647,6cyclictest1981794-21kworker/2:212:04:332
1947088995631,9cyclictest1981794-21kworker/2:212:22:572
194708499562,50cyclictest364-21snmpd07:54:341
194708499562,50cyclictest1947816-21munin-run07:25:001
194708499561,5cyclictest1948837-21sshd07:44:541
194708499561,50cyclictest364-21snmpd07:13:191
194707999564,3cyclictest131rcu_preempt12:02:530
194707999561,3cyclictest131rcu_preempt11:46:270
194707999561,3cyclictest131rcu_preempt11:31:160
194709399556,6cyclictest131rcu_preempt11:15:033
1947088995546,6cyclictest1939105-21kworker/2:109:20:002
1947088995546,5cyclictest1939105-21kworker/2:109:30:402
1947088995546,5cyclictest1939105-21kworker/2:109:30:392
1947088995545,6cyclictest1981794-21kworker/2:211:00:202
1947088995545,6cyclictest1981794-21kworker/2:211:00:202
1947088995531,9cyclictest1981794-21kworker/2:211:12:262
194708499553,48cyclictest1-21systemd08:15:001
194708499552,46cyclictest1949620-21munin-run08:00:011
194708499551,50cyclictest364-21snmpd08:18:331
194708499551,50cyclictest364-21snmpd08:18:331
194708499551,50cyclictest364-21snmpd07:38:241
1947079995529,18cyclictest2016921-21ssh11:24:040
1947079995527,13cyclictest2013233-21ssh11:16:090
194707999552,10cyclictest1962720-21grep09:30:010
1947088995446,5cyclictest1939105-21kworker/2:109:28:582
1947088995445,6cyclictest1986647-21kworker/2:311:47:112
194708499541,5cyclictest364-21snmpd07:46:311
194708499541,49cyclictest1951933-21modprobe08:40:541
194708499541,3cyclictest48-21kcompactd008:21:271
1947079995428,22cyclictest1957458-21ssh09:18:580
1947079995426,12cyclictest1959630-21ssh09:23:260
1947093995335,12cyclictest1-21systemd07:35:003
194708899537,32cyclictest1953256-21diskmemload11:55:032
1947088995345,5cyclictest1981794-21kworker/2:211:09:452
1947088995345,5cyclictest1981794-21kworker/2:210:31:152
1947088995345,5cyclictest1972273-21kworker/2:010:17:242
1947088995344,6cyclictest1981794-21kworker/2:210:57:402
1947088995342,6cyclictest1986647-21kworker/2:310:45:022
1947088995338,11cyclictest1-21systemd08:25:002
194707999535,32cyclictest131rcu_preempt09:34:020
194707999535,32cyclictest131rcu_preempt09:34:020
1947079995316,32cyclictest16850irq/134-800000010:43:470
1947079995316,31cyclictest16850irq/134-800000012:38:170
1947079995315,33cyclictest16850irq/134-800000009:46:350
1947093995237,12cyclictest37-21ksoftirqd/311:20:013
1947093995235,13cyclictest3467602-21systemd-journal09:20:003
1947088995244,5cyclictest1986647-21kworker/2:310:48:402
1947088995244,5cyclictest1939105-21kworker/2:107:20:012
1947088995243,6cyclictest1939105-21kworker/2:109:52:112
1947088995243,5cyclictest1981794-21kworker/2:210:30:012
194707999523,3cyclictest131rcu_preempt10:26:160
1947079995216,32cyclictest16850irq/134-800000010:45:260
1947079995216,31cyclictest16850irq/134-800000010:54:340
1947079995216,31cyclictest16850irq/134-800000010:54:330
1947079995215,32cyclictest16850irq/134-800000009:10:340
194707999521,36cyclictest2026590-21ssh11:44:260
194707999521,36cyclictest2026590-21ssh11:44:260
194709399513,8cyclictest37-21ksoftirqd/308:15:013
1947093995120,10cyclictest37-21ksoftirqd/312:02:043
1947088995143,5cyclictest1981794-21kworker/2:211:21:452
1947088995142,6cyclictest1939105-21kworker/2:109:36:512
1947088995142,5cyclictest1981794-21kworker/2:211:25:232
1947088995142,5cyclictest1972273-21kworker/2:010:11:412
1947079995116,32cyclictest16850irq/134-800000011:50:150
1947079995115,31cyclictest16850irq/134-800000009:50:380
1947079995115,31cyclictest16850irq/134-800000009:41:410
194707999511,3cyclictest131rcu_preempt12:22:370
194709399509,5cyclictest131rcu_preempt10:43:393
1947093995037,8cyclictest1953260-21munin-plugin-st09:10:003
1947093995034,10cyclictest1950666-21systemctl08:20:003
1947093995034,10cyclictest1950666-21systemctl08:20:003
1947093995033,12cyclictest309-21in:imuxsock09:10:043
1947093995032,15cyclictest37-21ksoftirqd/312:07:473
1947093995026,6cyclictest131rcu_preempt09:31:523
1947093995026,6cyclictest131rcu_preempt09:31:523
1947093995014,5cyclictest131rcu_preempt10:55:023
1947093995014,5cyclictest131rcu_preempt10:55:023
1947088995042,5cyclictest1986647-21kworker/2:310:23:372
1947088995042,5cyclictest1986647-21kworker/2:310:23:372
1947088995042,5cyclictest1981794-21kworker/2:211:41:562
1947088995042,5cyclictest1981794-21kworker/2:211:41:562
1947088995042,5cyclictest1939105-21kworker/2:110:00:032
1947088995041,6cyclictest1939105-21kworker/2:108:30:002
1947088995041,5cyclictest1939105-21kworker/2:109:41:202
1947088995041,5cyclictest1939105-21kworker/2:107:10:012
1947079995016,31cyclictest16850irq/134-800000010:33:560
1947079995016,30cyclictest16850irq/134-800000012:10:160
1947079995016,30cyclictest16850irq/134-800000012:10:160
194707999501,4cyclictest131rcu_preempt11:08:170
1947093994931,14cyclictest1-21systemd07:20:013
1947093994927,16cyclictest3467602-21systemd-journal08:35:013
1947093994911,6cyclictest131rcu_preempt10:38:353
1947088994942,4cyclictest1939105-21kworker/2:110:00:362
1947088994941,5cyclictest1981794-21kworker/2:211:34:172
1947088994941,5cyclictest1981794-21kworker/2:210:54:482
1947088994941,5cyclictest1981794-21kworker/2:210:54:482
1947088994940,5cyclictest1939105-21kworker/2:109:20:222
1947088994934,11cyclictest3467602-21systemd-journal09:50:002
194707999492,8cyclictest1948583-21munin-plugin-st07:39:590
1947093994837,8cyclictest37-21ksoftirqd/309:53:463
1947093994835,10cyclictest37-21ksoftirqd/311:54:453
1947093994833,11cyclictest1-21systemd10:35:013
1947088994840,5cyclictest1981794-21kworker/2:212:19:562
1947088994840,5cyclictest1939105-21kworker/2:107:12:552
1947088994839,6cyclictest1939105-21kworker/2:108:14:082
1947088994839,5cyclictest1981794-21kworker/2:212:36:402
1947088994838,5cyclictest1986647-21kworker/2:311:51:302
194709399478,5cyclictest131rcu_preempt12:33:263
1947093994729,13cyclictest37-21ksoftirqd/311:59:123
1947093994728,14cyclictest275-21dbus-daemon11:50:003
1947093994711,7cyclictest131rcu_preempt08:50:003
1947093994711,6cyclictest131rcu_preempt09:24:543
1947088994740,4cyclictest1939105-21kworker/2:109:12:502
1947088994739,5cyclictest1981794-21kworker/2:212:09:292
1947088994739,5cyclictest1981794-21kworker/2:211:38:562
1947088994739,5cyclictest1981794-21kworker/2:211:16:092
1947088994739,5cyclictest1939105-21kworker/2:108:38:362
1947088994739,5cyclictest1939105-21kworker/2:108:38:362
1947079994736,7cyclictest19350irq/34-mmc108:50:010
1947093994636,7cyclictest37-21ksoftirqd/312:11:463
1947093994636,7cyclictest37-21ksoftirqd/312:11:453
1947093994632,10cyclictest3467602-21systemd-journal10:30:013
1947093994614,6cyclictest131rcu_preempt10:16:583
1947093994612,6cyclictest131rcu_preempt11:30:013
1947088994629,13cyclictest1949595-21cron08:00:002
1947079994628,14cyclictest1950366-21sshd08:12:540
1947079994623,3cyclictest0-21swapper/008:00:010
194707999461,41cyclictest1947019-21latency_hist07:10:010
194709399459,5cyclictest131rcu_preempt09:41:143
1947093994537,6cyclictest37-21ksoftirqd/311:38:293
1947093994536,6cyclictest37-21ksoftirqd/311:00:023
1947093994535,7cyclictest37-21ksoftirqd/311:02:153
1947093994535,7cyclictest37-21ksoftirqd/311:02:153
1947093994533,9cyclictest37-21ksoftirqd/308:30:023
1947093994529,12cyclictest1948578-21cat07:40:003
194709399451,38cyclictest3467602-21systemd-journal08:40:003
194709399451,38cyclictest3467602-21systemd-journal08:40:003
1947093994510,6cyclictest131rcu_preempt10:00:473
1947088994537,4cyclictest1939105-21kworker/2:108:05:022
1947088994533,9cyclictest30-21ksoftirqd/207:30:022
194708499454,37cyclictest1947162-21cat07:10:011
1947079994530,11cyclictest309-21in:imuxsock08:05:010
1947079994529,11cyclictest1950246-21sort08:10:020
194709399445,6cyclictest131rcu_preempt12:22:453
1947093994434,7cyclictest37-21ksoftirqd/307:30:023
1947093994433,8cyclictest37-21ksoftirqd/309:05:023
1947093994433,7cyclictest37-21ksoftirqd/309:36:153
1947093994432,8cyclictest37-21ksoftirqd/311:05:203
1947093994431,9cyclictest37-21ksoftirqd/309:49:053
1947093994431,10cyclictest37-21ksoftirqd/312:18:243
1947093994425,13cyclictest3467602-21systemd-journal07:45:013
1947088994435,5cyclictest1939105-21kworker/2:109:00:002
1947088994410,5cyclictest131rcu_preempt08:50:002
1947079994425,13cyclictest275-21dbus-daemon08:40:020
1947079994425,13cyclictest275-21dbus-daemon08:40:010
194709399439,19cyclictest131rcu_preempt10:14:223
1947093994332,8cyclictest37-21ksoftirqd/310:07:173
1947093994329,11cyclictest37-21ksoftirqd/310:20:103
1947093994329,11cyclictest37-21ksoftirqd/310:20:103
194708899434,26cyclictest1939608-21kworker/u8:009:05:022
1947088994335,5cyclictest1939105-21kworker/2:108:18:332
1947088994335,5cyclictest1939105-21kworker/2:108:18:332
1947088994331,9cyclictest1948297-21sshd07:33:542
1947088994330,9cyclictest1949357-21munin-plugin-st07:55:002
194707999433,6cyclictest1-21systemd08:30:010
1947079994330,7cyclictest309-21in:imuxsock07:55:010
194707999431,3cyclictest131rcu_preempt07:20:020
194707999431,14cyclictest1952483-21munin-run08:55:010
194707999431,14cyclictest1952483-21munin-run08:55:010
194709399425,12cyclictest37-21ksoftirqd/311:32:203
1947093994232,7cyclictest37-21ksoftirqd/308:10:013
1947093994228,10cyclictest3467602-21systemd-journal08:00:003
1947093994215,6cyclictest131rcu_preempt12:26:403
194709399421,4cyclictest131rcu_preempt10:47:043
1947093994213,5cyclictest131rcu_preempt09:26:533
1947088994233,5cyclictest1939105-21kworker/2:107:45:012
1947079994236,3cyclictest16950irq/224-800000008:20:000
1947079994236,3cyclictest16950irq/224-800000008:20:000
194709399418,6cyclictest131rcu_preempt07:55:003
1947093994114,5cyclictest131rcu_preempt11:44:493
1947093994114,5cyclictest131rcu_preempt11:44:483
1947093994111,9cyclictest131rcu_preempt07:14:593
1947088994131,6cyclictest1939105-21kworker/2:107:25:002
1947088994116,5cyclictest30-21ksoftirqd/209:10:002
1947079994129,8cyclictest1947835-21latency_hist07:25:010
1947079994127,11cyclictest1947243-21sshd07:10:540
1947079994126,11cyclictest1951401-21sshd08:30:540
1947093994028,8cyclictest1949877-21cron08:05:003
1947079994026,10cyclictest364-21snmpd07:45:010
1947093993922,13cyclictest1952741-21tr09:00:003
194708899396,7cyclictest30-21ksoftirqd/207:40:022
1947088993930,6cyclictest1939105-21kworker/2:107:50:012
1947079993929,7cyclictest275-21dbus-daemon07:35:000
1947093993829,7cyclictest37-21ksoftirqd/308:54:543
1947093993829,7cyclictest37-21ksoftirqd/308:54:543
1947079993825,10cyclictest12-21ksoftirqd/008:22:580
194709399374,11cyclictest37-21ksoftirqd/308:23:243
1947093993723,10cyclictest3467602-21systemd-journal08:43:553
1947088993729,5cyclictest1939105-21kworker/2:108:05:002
1947079993724,9cyclictest12-21ksoftirqd/008:55:270
194707999372,30cyclictest1949252-21latency_hist07:50:020
194709399368,6cyclictest131rcu_preempt07:25:003
1947088993620,11cyclictest30-21ksoftirqd/208:30:422
1947079993617,16cyclictest16850irq/134-800000007:27:540
1947093993524,9cyclictest37-21ksoftirqd/307:10:013
1947088993423,7cyclictest1952443-21sshd08:52:542
1947088993423,7cyclictest1952443-21sshd08:52:542
1947079993424,7cyclictest12-21ksoftirqd/008:42:320
1947093993112,6cyclictest131rcu_preempt07:47:023
1947088993125,4cyclictest1939105-21kworker/2:108:43:372
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional