You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 12:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jan 12, 2026 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7112182910,6sleep013-21ksoftirqd/022:00:020
197399840,63rtkit-daemon0-21swapper/019:05:110
6966072830,5sleep30-21swapper/321:15:303
197399830,7rtkit-daemon927-21gmain22:35:202
197399830,7rtkit-daemon927-21gmain22:35:202
7437652800,6sleep10-21swapper/123:30:291
197399800,5rtkit-daemon0-21swapper/323:55:153
197399760,9rtkit-daemon1972-21rtkit-daemon21:49:522
197399740,9rtkit-daemon652494-21munin-node19:51:392
65047027354,13sleep10-21swapper/119:07:441
197399720,6rtkit-daemon4497-21PK-Backend00:24:563
197399720,5rtkit-daemon0-21swapper/323:51:353
197399720,5rtkit-daemon0-21swapper/323:51:353
65034626445,13sleep20-21swapper/219:06:082
65028126243,13sleep30-21swapper/319:05:283
197399610,9rtkit-daemon1972-21rtkit-daemon19:57:143
197399600,9rtkit-daemon1972-21rtkit-daemon20:16:593
197399590,9rtkit-daemon1972-21rtkit-daemon22:48:393
197399590,9rtkit-daemon1972-21rtkit-daemon20:26:542
7388712580,7chrt4497-21PK-Backend23:17:261
197399570,8rtkit-daemon1972-21rtkit-daemon19:41:453
7137322560,7chrt0-21swapper/222:05:232
65078899560,52cyclictest1972-21rtkit-daemon22:54:373
197399560,9rtkit-daemon1972-21rtkit-daemon22:21:282
650785995520,30cyclictest698746-21systemd-udevd21:23:402
650785995518,32cyclictest704412-21cat21:40:132
197399550,8rtkit-daemon1972-21rtkit-daemon22:41:112
197399540,9rtkit-daemon1972-21rtkit-daemon20:09:452
197399540,8rtkit-daemon1972-21rtkit-daemon21:50:062
197399540,8rtkit-daemon1972-21rtkit-daemon21:50:062
650785995316,32cyclictest736435-21ls23:10:202
65078899520,48cyclictest1972-21rtkit-daemon21:47:223
650785995217,30cyclictest710473-21cat21:55:282
650785995215,32cyclictest438-20systemd-journal19:45:022
650785995215,32cyclictest438-20systemd-journal19:45:022
650785995214,33cyclictest732650-21cat23:00:132
197399520,8rtkit-daemon1972-21rtkit-daemon22:30:412
650785995121,26cyclictest747813-21systemd23:45:012
650785995015,30cyclictest656199-21seq19:22:342
650785995015,30cyclictest656199-21seq19:22:342
650785994920,25cyclictest763620-21/usr/sbin/munin00:30:142
65078599482,41cyclictest923-21dbus-daemon20:40:012
650785994817,26cyclictest690668-21cstates21:00:122
650785994815,28cyclictest689074-21forks20:55:182
65078599481,42cyclictest668221-21sensors_temp19:55:272
650785994813,31cyclictest660535-21hddtemp_smartct19:35:192
650785994812,32cyclictest749911-21expr23:50:132
650785994812,32cyclictest749911-21expr23:50:132
650785994812,32cyclictest1033-21gdbus23:38:172
650785994812,31cyclictest730567-21cat22:55:132
7593192470,7sleep00-21swapper/000:15:280
7593192470,7sleep00-21swapper/000:15:270
65078599472,7cyclictest712738-21munin-run22:05:002
650785994718,25cyclictest763540-21sort00:30:012
650785994716,26cyclictest755428-21if_enp2s000:05:192
650785994714,28cyclictest734351-21expr23:05:142
650785994713,30cyclictest924-21NetworkManager00:00:022
650785994713,30cyclictest757356-21cat00:10:212
650785994712,30cyclictest748280-21expr23:45:142
650785994711,31cyclictest758453-21cpu00:15:122
650785994711,31cyclictest758453-21cpu00:15:122
197399470,29rtkit-daemon742817-21apt23:30:123
650785994617,25cyclictest686427-21turbostat.cron20:50:002
650785994614,28cyclictest721010-21vmstat22:25:322
650785994612,29cyclictest699343-21expr21:25:132
650785994611,31cyclictest683788-21grep20:40:222
650785994611,30cyclictest701160-21cat21:30:132
65078199466,5cyclictest141rcu_preempt22:35:221
65078199466,5cyclictest141rcu_preempt22:35:221
650777994617,25cyclictest13-21ksoftirqd/023:17:240
65078599459,13cyclictest727108-21cat22:45:152
650785994519,22cyclictest4497-21PK-Backend20:10:372
650785994519,22cyclictest4497-21PK-Backend20:10:372
650785994514,27cyclictest754301-21timerwakeupswit00:00:312
650785994513,27cyclictest658386-21acpi19:30:102
650785994416,24cyclictest743442-21missed_timers23:30:222
650785994413,27cyclictest653111-21latency19:15:062
650785994413,26cyclictest740307-21timerandwakeup23:20:292
650785994411,28cyclictest728748-21cpuspeed_turbos22:50:132
650785994411,28cyclictest702818-21ls21:35:142
650785994410,29cyclictest742010-21sort23:25:302
650785994410,29cyclictest742010-21sort23:25:302
650777994429,11cyclictest1033-21gdbus23:38:150
650785994318,21cyclictest1137-21runrttasks19:28:002
650785994317,22cyclictest716185-21dump-pmu-power22:19:562
650785994317,22cyclictest716185-21dump-pmu-power22:19:562
650785994315,25cyclictest663675-21latency_hist19:45:022
650785994315,23cyclictest760262-21cpuspeed00:20:132
650785994313,25cyclictest0-21swapper/221:15:152
650777994311,13cyclictest698724-21systemd-udevd21:23:400
197399430,29rtkit-daemon1918-21kerneloops20:56:520
650785994213,25cyclictest675820-21latency_hist20:20:012
650785994212,25cyclictest695582-21ls21:15:022
65078599421,12cyclictest716198-21perf22:15:002
650785994210,28cyclictest652439-21cron19:15:012
650785994210,11cyclictest679947-21interrupts20:30:192
650777994226,13cyclictest677083-21/usr/sbin/munin20:20:280
650777994222,16cyclictest752328-21processes23:55:270
197399420,35rtkit-daemon4497-21PK-Backend20:20:492
6840922410,4sleep30-21swapper/320:40:293
650788994123,13cyclictest438-20systemd-journal23:10:453
65078599419,10cyclictest738407-21processes23:15:262
65078599411,9cyclictest438-20systemd-journal20:05:002
650781994113,10cyclictest716203-21munin-run22:15:001
650777994124,12cyclictest715497-21awk22:10:250
197399410,27rtkit-daemon4497-21PK-Backend21:00:350
197399410,26rtkit-daemon0-21swapper/021:14:500
650788994020,15cyclictest2080-21gdbus23:02:063
65078599409,10cyclictest766396-21expr00:35:292
650777994022,14cyclictest13-21ksoftirqd/019:25:010
650777994022,14cyclictest13-21ksoftirqd/019:25:000
650777994021,7cyclictest13-21ksoftirqd/020:55:010
197399400,27rtkit-daemon4497-21PK-Backend20:27:523
650788993922,14cyclictest697726-21expr21:20:153
650788993919,16cyclictest653963-21ntp_states19:15:343
650781993925,10cyclictest740804-21cron23:25:001
650781993923,11cyclictest438-20systemd-journal20:10:001
650781993921,13cyclictest733559-21systemd-udevd23:02:051
650777993928,7cyclictest13-21ksoftirqd/000:05:000
650777993922,5cyclictest13-21ksoftirqd/023:05:230
650777993919,16cyclictest725314-21cstates22:40:130
65078899380,34cyclictest1972-21rtkit-daemon20:05:373
65078599389,25cyclictest6935632chrt21:09:052
65078599381,12cyclictest686910-21wc20:50:132
650777993821,14cyclictest13-21ksoftirqd/000:30:020
650777993820,14cyclictest13-21ksoftirqd/022:26:500
650788993723,10cyclictest673997-21cron20:15:003
650788993723,10cyclictest673997-21cron20:14:593
65078899371,19cyclictest6890502df20:55:153
65078199379,6cyclictest765216-21sed00:35:021
65078199372,31cyclictest678118-21diskstats20:25:171
650781993721,12cyclictest737033-21sshd23:11:451
65078199371,15cyclictest742548-21idleruntime-cro23:30:001
65078199371,15cyclictest742548-21idleruntime-cro23:30:001
65077799378,12cyclictest923-21dbus-daemon22:35:010
650777993725,8cyclictest680147-21grep20:30:220
65077799372,31cyclictest680936-21apt-get20:35:060
650777993722,12cyclictest13-21ksoftirqd/021:30:130
650777993719,7cyclictest13-21ksoftirqd/022:35:310
650777993719,7cyclictest13-21ksoftirqd/022:35:310
650777993719,14cyclictest13-21ksoftirqd/019:32:180
650777993719,13cyclictest438-20systemd-journal23:35:000
197399370,23rtkit-daemon0-21swapper/022:08:110
7498662360,6sleep00-21swapper/023:50:120
7498662360,6sleep00-21swapper/023:50:120
650788993623,8cyclictest734980-21python323:05:283
650788993623,10cyclictest665169-21runrttasks19:47:463
650788993623,10cyclictest665169-21runrttasks19:47:463
650788993622,11cyclictest42-21ksoftirqd/321:39:453
65078899362,13cyclictest713481-21irqstats22:05:193
650788993621,11cyclictest762983-21sed00:25:303
650788993621,11cyclictest759169-21ntp_kernel_pll_00:15:233
650788993621,11cyclictest759169-21ntp_kernel_pll_00:15:223
650788993620,12cyclictest1-21systemd19:35:003
650781993621,10cyclictest2069-21gnome-shell20:37:091
650781993619,13cyclictest708677-21python321:50:251
650781993619,13cyclictest708677-21python321:50:251
65078199361,6cyclictest4497-21PK-Backend19:50:001
65078199361,6cyclictest4497-21PK-Backend19:50:001
650777993624,6cyclictest13-21ksoftirqd/019:50:300
650777993619,5cyclictest13-21ksoftirqd/021:05:140
197399360,25rtkit-daemon0-21swapper/021:44:060
197399360,25rtkit-daemon0-21swapper/019:27:540
650788993528,5cyclictest42-21ksoftirqd/323:35:443
65078899352,29cyclictest660531-21/usr/sbin/munin19:35:183
650788993519,12cyclictest705287-21cat21:40:293
650788993519,12cyclictest1-21systemd19:15:013
650788993519,11cyclictest438-20systemd-journal22:43:453
650788993518,13cyclictest937-21polkitd23:17:233
650788993518,12cyclictest679527-21awk20:30:123
650788993517,13cyclictest438-20systemd-journal21:34:553
650788993515,16cyclictest749540-21systemd-run23:50:023
65078199356,14cyclictest141rcu_preempt20:45:101
65078199355,8cyclictest26-21ksoftirqd/119:20:141
65078199355,8cyclictest26-21ksoftirqd/119:20:141
650781993523,8cyclictest752179-21strings23:55:221
650781993518,13cyclictest651741-21ntpq19:10:251
65078199351,13cyclictest662582-21if_err_enp2s019:40:191
650777993525,6cyclictest675750-21turbostat20:20:000
650777993525,6cyclictest13-21ksoftirqd/020:40:210
650777993524,6cyclictest13-21ksoftirqd/023:02:080
650777993524,6cyclictest13-21ksoftirqd/021:50:000
650777993524,6cyclictest13-21ksoftirqd/021:25:170
650777993524,6cyclictest13-21ksoftirqd/019:35:280
650777993523,6cyclictest13-21ksoftirqd/021:15:280
65077799352,30cyclictest0-21swapper/000:24:010
650777993520,11cyclictest739520-21tr23:20:130
650777993519,12cyclictest13-21ksoftirqd/023:45:020
650777993518,12cyclictest766824-21cron00:40:000
650777993517,6cyclictest13-21ksoftirqd/021:50:100
650777993517,6cyclictest13-21ksoftirqd/021:50:100
650777993517,14cyclictest663251-40apt-check19:40:580
65077799351,30cyclictest718949-21ntpq22:20:260
65077799351,17cyclictest7287032tr22:50:130
197399350,25rtkit-daemon0-21swapper/022:01:410
197399350,24rtkit-daemon0-21swapper/300:11:063
65078899347,23cyclictest0-21swapper/321:50:453
65078899347,23cyclictest0-21swapper/321:50:453
650788993422,8cyclictest746718-21hddtemp_smartct23:40:183
65078899342,12cyclictest764043-21hddtemp_smartct00:30:193
65078899342,12cyclictest753635-21diskstats00:00:183
650788993420,10cyclictest721447-21cat22:30:013
650781993423,6cyclictest26-21ksoftirqd/120:02:561
65078199342,15cyclictest0-21swapper/121:10:011
650781993419,11cyclictest710261-21awk21:55:221
650781993419,11cyclictest684800-21sed20:45:011
650781993418,11cyclictest761889-21find00:25:111
65078199341,28cyclictest4497-21PK-Backend00:20:081
650777993423,8cyclictest13-21ksoftirqd/020:50:010
650777993423,7cyclictest671491-21needreboot20:05:230
650777993423,6cyclictest13-21ksoftirqd/023:45:120
650777993423,6cyclictest13-21ksoftirqd/022:45:200
650777993423,6cyclictest13-21ksoftirqd/020:10:140
650777993423,6cyclictest13-21ksoftirqd/020:10:130
650777993423,6cyclictest13-21ksoftirqd/019:15:030
650777993423,6cyclictest13-21ksoftirqd/000:10:010
650777993423,5cyclictest13-21ksoftirqd/019:10:200
650777993422,9cyclictest13-21ksoftirqd/022:15:290
650777993422,9cyclictest13-21ksoftirqd/022:15:290
650777993419,10cyclictest438-20systemd-journal20:02:450
650777993418,12cyclictest762634-21ntpq00:25:240
650777993417,6cyclictest13-21ksoftirqd/019:45:130
650777993417,6cyclictest13-21ksoftirqd/019:45:120
65078899339,7cyclictest0-21swapper/321:10:123
65078899338,12cyclictest141rcu_preempt23:20:103
650788993324,6cyclictest693659-21munin-plugin-st21:10:003
650788993322,7cyclictest691540-21users21:00:293
65078899332,27cyclictest666195-21sed19:50:213
65078899332,12cyclictest657476-21ps19:25:273
650788993319,10cyclictest715717-21grep22:10:293
650788993319,10cyclictest669282-21expr20:00:143
650788993317,12cyclictest730447-21grep22:55:113
650788993317,11cyclictest682340-21python320:35:303
65078899331,29cyclictest4497-21PK-Backend22:22:153
65078899331,29cyclictest4497-21PK-Backend00:37:393
65078899331,27cyclictest699130-21/usr/sbin/munin21:25:103
650781993325,4cyclictest4497-21PK-Backend19:59:201
650781993324,5cyclictest4497-21PK-Backend00:10:081
650781993322,7cyclictest717115-21needreboot22:15:231
650781993322,7cyclictest717115-21needreboot22:15:231
650781993319,10cyclictest688182-21dpkg20:55:001
650781993318,10cyclictest718594-21seq22:20:181
650781993317,11cyclictest694870-21timerandwakeup21:10:281
650781993316,12cyclictest673502-21grep20:10:311
650781993316,12cyclictest673502-21grep20:10:311
65078199331,5cyclictest4497-21PK-Backend00:01:221
650781993315,13cyclictest680936-21apt-get20:35:011
65078199331,29cyclictest4497-21PK-Backend23:42:361
65078199331,29cyclictest4497-21PK-Backend21:49:431
65078199331,29cyclictest0-21swapper/123:35:251
65078199331,12cyclictest1137-21runrttasks21:35:271
65077799335,24cyclictest758194-21turbostat00:15:000
65077799334,10cyclictest13-21ksoftirqd/022:55:190
650777993323,6cyclictest13-21ksoftirqd/021:40:020
650777993317,12cyclictest923-21dbus-daemon20:00:000
650777993317,12cyclictest736570-21ntpq23:10:220
65077799331,4cyclictest4497-21PK-Backend23:25:290
65077799331,4cyclictest4497-21PK-Backend23:25:290
65077799331,30cyclictest0-21swapper/020:27:170
65078899322,26cyclictest722399-21ntp_states22:30:253
650788993222,6cyclictest42-21ksoftirqd/323:25:243
650788993222,6cyclictest42-21ksoftirqd/323:25:243
650788993218,10cyclictest716882-21basename22:15:203
650788993218,10cyclictest716882-21basename22:15:203
650788993218,10cyclictest711792-21grep22:00:213
650788993218,10cyclictest42-21ksoftirqd/300:05:143
650788993216,12cyclictest655016-21cat19:20:143
650788993216,12cyclictest655016-21cat19:20:143
65078899321,4cyclictest4497-21PK-Backend20:46:143
65078899321,4cyclictest0-21swapper/321:56:553
650781993225,4cyclictest4497-21PK-Backend21:42:501
650781993225,4cyclictest4497-21PK-Backend21:03:451
650781993225,4cyclictest0-21swapper/119:15:141
650781993224,5cyclictest4497-21PK-Backend22:51:381
650781993222,6cyclictest659861-21dump-pmu-power19:35:001
650781993222,6cyclictest26-21ksoftirqd/120:22:591
65078199322,12cyclictest0-21swapper/121:15:161
65078199322,10cyclictest141rcu_preempt22:33:381
650781993217,11cyclictest712157-21switchtime22:00:301
65078199321,4cyclictest4497-21PK-Backend22:45:421
650781993213,15cyclictest4497-21PK-Backend00:10:001
65078199321,29cyclictest0-21swapper/120:16:131
65078199321,28cyclictest4497-21PK-Backend22:26:371
65078899311,10cyclictest723714-21cstates22:35:163
65078899311,10cyclictest723714-21cstates22:35:163
65078199314,6cyclictest141rcu_preempt23:47:581
650781993124,4cyclictest4497-21PK-Backend19:28:411
650781993122,5cyclictest26-21ksoftirqd/121:25:151
650781993121,6cyclictest26-21ksoftirqd/120:55:281
650781993117,10cyclictest765427-21awk00:35:131
650781993116,11cyclictest751439-21ls23:55:011
650781993116,11cyclictest751439-21ls23:55:011
650781993116,11cyclictest731259-21timerandwakeup22:55:271
65078199311,4cyclictest4497-21PK-Backend22:44:081
65078199311,4cyclictest4497-21PK-Backend22:05:211
65078899301,4cyclictest4497-21PK-Backend20:51:333
650781993023,4cyclictest4497-21PK-Backend21:31:531
65078199301,6cyclictest4497-21PK-Backend21:23:391
650781993015,5cyclictest141rcu_preempt19:55:001
650781992915,10cyclictest735573-21ls23:10:001
65078199291,24cyclictest759923-21basename00:20:001
65078199291,24cyclictest759923-21basename00:20:001
650788992816,8cyclictest6771952sleep320:21:373
65078199286,10cyclictest141rcu_preempt19:36:291
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional