You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 09:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 29, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162399960,22rtkit-daemon0-21swapper/019:47:510
162399920,4rtkit-daemon0-21swapper/121:33:161
162399870,5rtkit-daemon0-21swapper/100:25:321
162399860,5rtkit-daemon0-21swapper/123:12:191
162399800,5rtkit-daemon0-21swapper/122:59:581
19282942790,5chrt1928299-21cron21:20:001
188205827941,32sleep10-21swapper/119:07:151
188213127551,18sleep20-21swapper/219:08:132
162399750,5rtkit-daemon0-21swapper/123:03:111
162399740,5rtkit-daemon0-21swapper/123:05:551
188213426944,16sleep00-21swapper/019:08:140
188214726641,15sleep30-21swapper/319:08:253
1882437995920,16cyclictest1939805-21cstates21:50:151
19364282580,6chrt2197014-21PK-Backend21:40:150
19012202560,10sleep21753-21gnome-shell20:00:232
1882437995613,20cyclictest438-20systemd-journal23:30:001
188243799551,16cyclictest1945194-21hddtemp_smartct22:05:181
19097082540,15sleep2331rcuc/220:25:142
188243799541,15cyclictest1938726-21sensors_temp21:45:281
1882435995438,6cyclictest13-21ksoftirqd/023:23:300
188243799532,28cyclictest1753-21gnome-shell21:40:151
188243799521,28cyclictest1908485-21cat20:20:331
1882442995022,9cyclictest1895386-21grep19:45:143
188243799502,22cyclictest1-21systemd00:15:001
1882437995013,15cyclictest1753-21gnome-shell20:33:101
188243799501,17cyclictest1753-21gnome-shell20:10:161
1882435995010,9cyclictest13-21ksoftirqd/021:30:040
162399500,36rtkit-daemon1897195-21ls19:50:150
1882442994919,26cyclictest1957996-21uname22:40:263
188243799491,17cyclictest1909421-21grep20:25:141
1882437994910,17cyclictest1924406-21timerwakeupswit21:05:301
188243799482,15cyclictest1893272-21munin-run19:40:021
1882437994820,23cyclictest1975661-21awk23:30:261
188243799481,25cyclictest1983922-21df23:55:161
188243799481,25cyclictest1983922-21df23:55:151
19680492470,8chrt42-21ksoftirqd/323:10:213
188243799472,24cyclictest1955269-21munin-run22:35:011
188243799472,23cyclictest875-21polkitd00:30:321
188243799472,20cyclictest1753-21gnome-shell22:40:191
188243799471,24cyclictest1902050-21sed20:05:031
188243799471,20cyclictest1987456-21hddtemp_smartct00:05:191
188243799471,20cyclictest1955574-21cat22:35:151
188243799471,20cyclictest1899713-21wc19:55:301
188243799471,18cyclictest1753-21gnome-shell20:15:191
188243799471,16cyclictest1902882-21needreboot20:05:221
1882435994723,19cyclictest1935562-21munin-run21:40:010
1882435994718,8cyclictest13-21ksoftirqd/000:30:300
19399882460,8sleep20-21swapper/221:50:192
188243799462,18cyclictest1753-21gnome-shell23:45:241
188243799461,20cyclictest1982202-21diskstats23:50:171
1882437994531,11cyclictest3242254-21gmain23:45:001
188243799451,27cyclictest1884370-21irqstats19:15:041
188243799451,22cyclictest1753-21gnome-shell20:41:561
188243799451,20cyclictest1949569-21taskset22:18:091
188243799451,19cyclictest1943934-21python322:00:291
188243799451,18cyclictest1941379-21/usr/sbin/munin21:55:121
188243799451,18cyclictest1929223-21cat21:20:221
188243799451,12cyclictest1962175-21munin-run22:55:001
1882435994531,10cyclictest1965669-21unin-run23:05:000
188243599452,16cyclictest13-21ksoftirqd/022:44:590
162399450,31rtkit-daemon438-20systemd-journal20:35:000
19356042440,7sleep3411rcuc/321:40:023
188244299449,16cyclictest1753-21gnome-shell19:54:373
1882442994417,22cyclictest1976356-21cron23:35:003
1882442994412,28cyclictest1932917-40lsb_release21:30:243
1882442994410,30cyclictest1753-21gnome-shell00:01:023
1882442994410,30cyclictest1753-21gnome-shell00:01:023
188243799442,25cyclictest1998204-21hddtemp_smartct00:35:191
188243799441,27cyclictest1890792-21ntp_states19:30:261
188243799441,25cyclictest1914121-21seq20:36:071
188243799441,18cyclictest1925477-21df_inode21:10:151
188243799441,16cyclictest0-21swapper/122:10:161
188243799441,14cyclictest1897722-21ntp_states19:50:261
1882435994421,14cyclictest13-21ksoftirqd/020:35:150
162399440,8rtkit-daemon1753-21gnome-shell19:33:080
19414402430,7sleep02197014-21PK-Backend21:55:130
1882442994325,13cyclictest1999296-21pool-org.gnome.00:40:003
1882440994334,6cyclictest356-21jbd2/sdb2-800:00:022
1882440994334,6cyclictest356-21jbd2/sdb2-800:00:022
188243799433,15cyclictest1950418-21basename22:20:201
188243799432,17cyclictest1895620-21diskstats19:45:171
188243799432,17cyclictest1753-21gnome-shell23:20:051
188243799431,17cyclictest1753-21gnome-shell21:28:511
188243799431,16cyclictest1882977-21df19:10:171
188243799431,11cyclictest1986818-21munin-run00:05:011
188243799431,11cyclictest1986818-21munin-run00:05:011
188243599436,18cyclictest1753-21gnome-shell00:00:230
188243599436,18cyclictest1753-21gnome-shell00:00:220
188244299428,30cyclictest1753-21gnome-shell22:01:543
1882442994222,15cyclictest3242253-21nautilus21:30:033
1882442994212,11cyclictest2181-21gdbus23:05:003
188244099428,12cyclictest141rcu_preempt21:55:222
1882440994219,18cyclictest1951447-21cron22:24:592
1882440994211,6cyclictest141rcu_preempt23:30:002
188243799421,18cyclictest1935470-21runrttasks21:39:091
188243799421,15cyclictest1991264-21ntp_states00:15:261
1882435994227,11cyclictest1928296-21cron21:19:590
1882442994127,10cyclictest1753-21gnome-shell23:57:403
1882442994127,10cyclictest1753-21gnome-shell23:57:403
1882442994126,11cyclictest42-21ksoftirqd/320:15:283
1882442994119,17cyclictest42-21ksoftirqd/323:05:233
1882442994118,11cyclictest42-21ksoftirqd/322:50:183
1882442994116,13cyclictest1989018-21cat00:10:153
188244299411,36cyclictest1753-21gnome-shell23:38:573
188244299411,36cyclictest1753-21gnome-shell23:38:573
188244099415,11cyclictest141rcu_preempt00:09:202
188244099412,34cyclictest1753-21gnome-shell21:48:302
188244099412,18cyclictest1753-21gnome-shell20:47:392
188244099411,9cyclictest2197014-21PK-Backend23:45:482
1882435994130,7cyclictest1753-21gnome-shell22:35:540
1882435994129,9cyclictest13-21ksoftirqd/000:00:010
1882435994129,9cyclictest13-21ksoftirqd/000:00:010
1882435994129,7cyclictest438-20systemd-journal21:24:560
1882435994123,5cyclictest13-21ksoftirqd/020:40:200
1882435994122,15cyclictest13-21ksoftirqd/022:32:010
1882435994122,15cyclictest13-21ksoftirqd/000:10:300
1882435994121,5cyclictest13-21ksoftirqd/023:42:300
1882435994119,18cyclictest1943666-21missed_timers22:00:220
188243599411,8cyclictest1753-21gnome-shell22:55:240
188244299409,7cyclictest141rcu_preempt19:58:553
1882442994015,21cyclictest1995991-40apt-check00:30:173
1882442994013,8cyclictest1753-21gnome-shell20:10:253
1882440994029,7cyclictest1912166-21switchtime20:30:322
1882440994024,11cyclictest1931774-21apt-get21:30:002
188244099402,34cyclictest1894985-21munin-run19:45:002
1882440994023,13cyclictest1945275-21irqstats22:05:212
188244099401,34cyclictest1895218-21/usr/sbin/munin19:45:182
188244099401,12cyclictest1908599-21taskset20:21:372
188243799402,15cyclictest1993008-21ntp_states00:20:271
188243799401,16cyclictest1889050-21ntp_states19:25:241
188243799401,15cyclictest1959455-21irqstats22:45:201
188243799401,15cyclictest1952256-21expr22:25:131
188243599403,15cyclictest1949786-21cut22:20:030
1882435994024,6cyclictest13-21ksoftirqd/000:24:430
1882435994023,6cyclictest13-21ksoftirqd/019:45:030
1882435994022,7cyclictest13-21ksoftirqd/020:58:130
1882435994022,14cyclictest13-21ksoftirqd/020:45:510
1882435994022,14cyclictest13-21ksoftirqd/019:15:230
188243599401,6cyclictest141rcu_preempt23:11:310
1882435994015,20cyclictest1990897-21diskstats00:15:170
188243599401,18cyclictest1888034-21perf19:29:550
1882442993928,7cyclictest1753-21gnome-shell22:22:183
1882442993925,11cyclictest1995461-21awk00:30:013
1882442993924,11cyclictest42-21ksoftirqd/321:20:173
188244299392,33cyclictest1753-21gnome-shell22:19:283
188244299392,33cyclictest1753-21gnome-shell22:07:353
188244299392,33cyclictest1753-21gnome-shell00:18:183
1882442993923,12cyclictest1908880-21apt-get20:25:003
1882442993916,20cyclictest391irq_work/320:09:043
1882442993911,11cyclictest42-21ksoftirqd/322:33:593
1882440993928,7cyclictest884-21gdbus00:30:302
188244099392,33cyclictest1753-21gnome-shell21:36:382
1882440993921,6cyclictest34-21ksoftirqd/221:44:522
188244099391,34cyclictest1975748-21rt-features23:30:282
1882437993914,20cyclictest1969810-21if_enp2s023:15:191
188243799391,16cyclictest1976989-21df23:35:171
188243799391,16cyclictest1976989-21df23:35:161
188243799391,15cyclictest1923108-21munin-run21:05:001
188243599395,9cyclictest13-21ksoftirqd/021:10:000
188243599395,7cyclictest141rcu_preempt20:05:010
1882435993928,7cyclictest1753-21gnome-shell20:21:270
1882435993922,7cyclictest13-21ksoftirqd/022:48:090
1882435993921,7cyclictest13-21ksoftirqd/023:51:000
1882435993920,7cyclictest13-21ksoftirqd/023:25:040
188243599391,34cyclictest1947986-21turbostat22:15:000
188244299387,5cyclictest141rcu_preempt21:45:213
188244299387,11cyclictest42-21ksoftirqd/319:28:203
1882442993831,4cyclictest0-21swapper/320:45:023
1882442993826,8cyclictest1971272-21tr23:20:163
1882442993820,13cyclictest1901066-21memory20:00:213
188244299381,9cyclictest2197014-21PK-Backend21:55:013
1882442993818,16cyclictest1753-21gnome-shell00:05:163
1882440993825,8cyclictest1753-21gnome-shell00:25:082
188244099382,32cyclictest1884088-21munin-run19:15:002
1882440993822,5cyclictest34-21ksoftirqd/219:18:092
1882440993821,5cyclictest34-21ksoftirqd/200:03:012
1882440993821,5cyclictest34-21ksoftirqd/200:03:012
1882440993821,13cyclictest1904394-21if_enp2s020:10:202
1882440993820,6cyclictest34-21ksoftirqd/223:50:202
1882440993820,6cyclictest34-21ksoftirqd/223:23:572
1882440993817,17cyclictest855-21dbus-daemon00:40:022
188244099381,5cyclictest2197014-21PK-Backend23:40:012
188244099381,5cyclictest2197014-21PK-Backend23:40:012
1882440993814,6cyclictest141rcu_preempt19:51:012
1882437993820,14cyclictest1887325-21ntp_states19:20:271
1882437993820,14cyclictest1887325-21ntp_states19:20:271
188243799381,16cyclictest1921380-21munin-run21:00:001
188243799381,15cyclictest1893503-21apt19:40:081
188243599388,5cyclictest141rcu_preempt20:30:020
188243599382,7cyclictest1753-21gnome-shell22:54:070
1882435993827,7cyclictest1753-21gnome-shell23:35:290
1882435993827,7cyclictest1753-21gnome-shell23:35:290
1882435993827,7cyclictest1753-21gnome-shell23:09:390
1882435993827,7cyclictest1753-21gnome-shell20:52:590
1882435993827,7cyclictest1753-21gnome-shell00:36:530
1882435993827,7cyclictest1753-21gnome-shell00:07:080
1882435993821,6cyclictest13-21ksoftirqd/019:12:320
1882435993820,7cyclictest13-21ksoftirqd/023:18:410
1882435993820,7cyclictest13-21ksoftirqd/021:25:300
1882435993811,8cyclictest1976181-21sshd23:32:570
188243599381,18cyclictest13-21ksoftirqd/021:49:140
188243599381,12cyclictest1951801-21latency_hist22:25:030
18938352370,6sleep3411rcuc/319:40:133
188244299378,18cyclictest1753-21gnome-shell19:39:503
1882442993728,5cyclictest411rcuc/323:25:133
188244299372,7cyclictest1753-21gnome-shell21:55:433
1882442993726,7cyclictest1963163-21grep22:55:263
188244299372,32cyclictest0-21swapper/322:35:253
188244299372,30cyclictest1910653-40lsb_release20:27:193
1882442993722,11cyclictest42-21ksoftirqd/323:50:123
188244299372,16cyclictest1753-21gnome-shell23:49:563
1882442993721,12cyclictest1925830-21ntpq21:10:233
1882442993720,6cyclictest42-21ksoftirqd/321:15:303
1882442993720,6cyclictest1992613-21expr00:20:163
1882442993720,13cyclictest42-21ksoftirqd/323:19:113
188244099377,6cyclictest141rcu_preempt19:30:122
1882440993726,7cyclictest1955236-21sh22:35:012
1882440993725,9cyclictest1795-21gmain22:50:012
1882440993720,12cyclictest1923534-21cut21:05:132
188243799371,15cyclictest1918371-21cstates20:50:151
1882437993710,22cyclictest1753-21gnome-shell20:45:141
1882435993726,7cyclictest1753-21gnome-shell21:52:170
1882435993726,7cyclictest1753-21gnome-shell20:17:270
1882435993726,7cyclictest1753-21gnome-shell20:13:290
1882435993726,7cyclictest1753-21gnome-shell20:09:390
1882435993726,7cyclictest1753-21gnome-shell19:57:180
1882435993726,7cyclictest1753-21gnome-shell19:38:060
1882435993726,7cyclictest1753-21gnome-shell19:20:210
1882435993726,7cyclictest1753-21gnome-shell19:20:200
1882435993720,13cyclictest13-21ksoftirqd/022:27:070
1882435993719,5cyclictest13-21ksoftirqd/021:00:220
1882435993712,8cyclictest1946265-21sh22:10:000
188244299364,11cyclictest42-21ksoftirqd/320:31:433
1882442993622,10cyclictest42-21ksoftirqd/323:41:293
1882442993620,12cyclictest1952085-21cpuspeed_turbos22:25:133
1882442993619,12cyclictest904-21systemd-logind22:46:503
1882442993618,14cyclictest1753-21gnome-shell20:50:193
1882442993618,13cyclictest438-20systemd-journal20:55:263
188244299361,31cyclictest1936834-21ntp_states21:40:263
188244099362,9cyclictest1961084-21diskstats22:50:172
1882440993621,11cyclictest1927583-21ntpq21:15:242
188244099361,17cyclictest1069-21runrttasks22:59:212
188244099361,13cyclictest2197014-21PK-Backend22:19:072
188244099361,13cyclictest2197014-21PK-Backend19:40:002
188244099361,12cyclictest2197014-21PK-Backend20:51:372
188244099361,10cyclictest1965993-21/usr/sbin/munin23:05:132
1882435993617,7cyclictest13-21ksoftirqd/000:27:150
1882442993523,8cyclictest1753-21gnome-shell20:39:333
1882442993523,8cyclictest1753-21gnome-shell19:17:573
188244299351,6cyclictest2197014-21PK-Backend22:10:233
188244299351,31cyclictest2197014-21PK-Backend20:47:323
188244299351,12cyclictest1753-21gnome-shell21:00:393
188244099356,10cyclictest34-21ksoftirqd/220:43:272
1882440993524,7cyclictest1753-21gnome-shell00:23:082
188244099351,9cyclictest2197014-21PK-Backend20:36:402
1882440993519,5cyclictest34-21ksoftirqd/222:02:482
1882440993519,12cyclictest3242254-21gmain19:30:012
1882440993518,5cyclictest34-21ksoftirqd/220:55:492
1882440993518,5cyclictest34-21ksoftirqd/200:10:592
1882440993516,15cyclictest3242253-21nautilus23:19:062
188244099351,14cyclictest2197014-21PK-Backend00:17:252
1882435993525,6cyclictest1753-21gnome-shell23:47:160
19582042340,8chrt2197014-21PK-Backend22:40:302
188244299342,7cyclictest2197014-21PK-Backend19:34:273
1882442993422,8cyclictest1753-21gnome-shell19:22:563
1882442993422,8cyclictest1753-21gnome-shell19:22:563
188244299342,14cyclictest1753-21gnome-shell19:10:233
1882442993419,11cyclictest1923648-21ls21:05:153
1882440993421,9cyclictest1753-21gnome-shell21:21:482
1882440993420,10cyclictest34-21ksoftirqd/221:10:122
188244099341,9cyclictest2197014-21PK-Backend23:11:032
188244099341,9cyclictest2197014-21PK-Backend21:00:402
1882440993418,12cyclictest1932701-21cat21:30:182
188244099341,13cyclictest2197014-21PK-Backend22:39:572
1882435993423,6cyclictest13-21ksoftirqd/021:11:380
1882440993325,5cyclictest2197014-21PK-Backend23:02:382
1882440993321,8cyclictest1902275-21cpuspeed20:05:122
188244099331,9cyclictest2197014-21PK-Backend20:18:262
188244099331,9cyclictest2197014-21PK-Backend19:55:342
1882440993317,12cyclictest1947278-21sshd22:10:262
1882440993218,11cyclictest1753-21gnome-shell22:27:262
1882440993210,18cyclictest1753-21gnome-shell23:40:322
188244099319,18cyclictest1753-21gnome-shell19:23:262
188244099319,18cyclictest1753-21gnome-shell19:23:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional