You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-21 - 12:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Wed May 21, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
268839199753749,3cyclictest2920871-21kworker/3:1+events23:02:303
268839199753749,3cyclictest2809081-21kworker/3:2+events21:42:583
268839199752748,3cyclictest2920871-21kworker/3:1+events22:22:333
268839199751747,3cyclictest2809081-21kworker/3:2+events21:24:263
268839199750746,3cyclictest2740217-21kworker/3:0+events19:59:033
268839199750745,4cyclictest2809081-21kworker/3:2+events21:27:533
268839199749745,3cyclictest2740217-21kworker/3:0+events20:09:153
268839199748744,3cyclictest2970756-21kworker/3:2+events00:23:013
268839199748744,3cyclictest2809081-21kworker/3:2+events21:45:223
268839199747743,3cyclictest2920871-21kworker/3:1+events22:42:533
268839199746740,5cyclictest2970756-21kworker/3:2+events00:10:423
268839199746737,4cyclictest2740217-21kworker/3:0+events20:10:433
268839199745737,5cyclictest2970756-21kworker/3:2+events23:32:553
268839199745737,5cyclictest2809081-21kworker/3:2+events21:36:133
268839199744740,3cyclictest2970756-21kworker/3:2+events23:28:583
268839199744740,3cyclictest2809081-21kworker/3:2+events20:46:163
268839199744739,4cyclictest2809081-21kworker/3:2+events21:34:273
268839199744739,4cyclictest2649389-21kworker/3:2+events19:13:313
268839199744738,5cyclictest2920871-21kworker/3:1+events22:32:533
268838599744738,5cyclictest2902598-21kworker/1:1+events23:57:411
268839199743740,2cyclictest2920871-21kworker/3:1+events22:52:103
268839199743740,2cyclictest2920871-21kworker/3:1+events22:52:103
268839199743739,3cyclictest2920871-21kworker/3:1+events22:36:493
268839199743739,3cyclictest2809081-21kworker/3:2+events21:01:073
268839199743739,3cyclictest2740217-21kworker/3:0+events20:03:143
268839199743738,4cyclictest2920871-21kworker/3:1+events22:27:483
268839199743737,5cyclictest2970756-21kworker/3:2+events00:05:213
268839199743736,6cyclictest2970756-21kworker/3:2+events00:30:213
268839199742738,3cyclictest2970756-21kworker/3:2+events23:37:273
268839199742738,3cyclictest2970756-21kworker/3:2+events00:18:243
268839199742738,3cyclictest2920871-21kworker/3:1+events22:55:453
268839199742738,3cyclictest2809081-21kworker/3:2+events21:08:033
268838799742737,4cyclictest2694632-21kworker/2:3+events19:30:232
268839199741737,3cyclictest2809081-21kworker/3:2+events21:52:003
268839199741737,3cyclictest2809081-21kworker/3:2+events21:13:153
268839199740736,3cyclictest2970756-21kworker/3:2+events00:38:563
268839199740736,3cyclictest2688156-21kworker/3:1+events19:17:113
268838599740735,4cyclictest2902598-21kworker/1:1+events23:49:541
268839199739735,3cyclictest2920871-21kworker/3:1+events22:17:103
268839199739735,3cyclictest2809081-21kworker/3:2+events22:09:413
268839199739734,4cyclictest2920871-21kworker/3:1+events22:14:053
268839199739734,4cyclictest2809081-21kworker/3:2+events21:18:053
268839199738734,3cyclictest2920871-21kworker/3:1+events23:07:093
268839199738734,3cyclictest2896321-21turbostat21:59:563
268839199738734,3cyclictest2809081-21kworker/3:2+events22:02:183
268838599738734,3cyclictest2649695-21kworker/1:0+events19:22:451
268839199736732,3cyclictest2920871-21kworker/3:1+events22:47:473
268839199736731,4cyclictest2809081-21kworker/3:2+events20:50:533
268839199736731,4cyclictest2809081-21kworker/3:2+events20:50:533
268838599736731,4cyclictest2902598-21kworker/1:1+events00:03:271
268839199735731,3cyclictest2970756-21kworker/3:2+events00:28:343
268838599735731,3cyclictest2902598-21kworker/1:1+events23:44:021
268839199734730,3cyclictest2809081-21kworker/3:2+events20:57:093
268838599732728,3cyclictest2902598-21kworker/1:1+events23:52:571
268838799730726,3cyclictest2694632-21kworker/2:3+events19:48:192
268838799730723,6cyclictest2694632-21kworker/2:3+events19:36:482
268839199729725,3cyclictest2920871-21kworker/3:1+events23:14:083
268838799728723,4cyclictest2694632-21kworker/2:3+events20:24:012
268838799727723,3cyclictest2694632-21kworker/2:3+events19:51:072
268838799726723,2cyclictest2694632-21kworker/2:3+events20:43:552
268838799724721,2cyclictest2694632-21kworker/2:3+events21:57:432
268838799723718,4cyclictest2694632-21kworker/2:3+events20:32:342
268838799722719,2cyclictest2694632-21kworker/2:3+events20:36:502
268838799722719,2cyclictest2694632-21kworker/2:3+events19:28:262
268838799719716,2cyclictest2694632-21kworker/2:3+events20:17:312
268838799719715,3cyclictest2694632-21kworker/2:3+events23:23:322
268838799719715,3cyclictest2694632-21kworker/2:3+events19:42:212
268838799718714,3cyclictest2694632-21kworker/2:3+events20:26:042
268838799715710,4cyclictest2694632-21kworker/2:3+events23:16:362
268839199703699,3cyclictest3052679-21turbostat00:04:553
268839199693688,4cyclictest2740187-21turbostat19:54:563
268838599689685,3cyclictest2694546-21turbostat19:19:551
268838799687681,5cyclictest2702896-21turbostat19:24:552
268838799673670,2cyclictest2890093-21turbostat21:54:552
268839199671667,3cyclictest2802821-21turbostat20:44:553
268839199666663,2cyclictest3002425-21turbostat23:24:553
268838799647644,2cyclictest2989732-21turbostat23:14:562
268838599644641,2cyclictest3021290-21turbostat23:39:551
268838799602599,2cyclictest2765099-21turbostat20:14:552
27915422380,1sleep10-21swapper/120:30:171
26923132350,2sleep1301rcuc/119:10:181
29183582280,1chrt2918361-21cut22:10:212
27717592280,1chrt2771758-21df20:15:163
2688391992724,2cyclictest3052721-21systemd-run00:00:013
2688381992119,1cyclictest2777611-21idleruntime-cro20:20:010
2688381992118,2cyclictest523-21gdbus00:15:000
2688385992016,2cyclictest0-21swapper/119:45:011
2688387991916,2cyclictest3077916-21cron00:20:002
2688387991915,2cyclictest0-21swapper/200:40:002
2688385991915,2cyclictest0-21swapper/119:30:001
268838199198,2cyclictest2859015-21turbostat21:25:000
2688387991814,3cyclictest506-21in:imuxsock22:05:002
2688385991814,2cyclictest0-21swapper/121:35:011
2688381991812,4cyclictest3028177-21cut23:40:150
2688391991713,3cyclictest0-21swapper/319:30:003
2688391991712,4cyclictest3614154-21systemd-journal19:45:013
2688387991714,2cyclictest3052659-21latency_hist00:00:012
2688387991714,2cyclictest2882970-21needreboot21:40:202
2688387991712,3cyclictest2847836-21irqrtprio21:15:212
2688391991612,3cyclictest2797190-21gltestperf20:35:153
2688387991612,2cyclictest0-21swapper/223:40:012
2688387991611,3cyclictest2867905-21kthreadcore21:30:202
2688385991613,2cyclictest3014810-21cron23:30:001
2688385991612,3cyclictest2721203-21unixbench_singl19:30:241
2688385991612,2cyclictest2920742-21tail22:10:281
2688385991611,3cyclictest0-21swapper/122:10:001
268838199168,6cyclictest493-21dbus-daemon21:55:010
2688381991614,1cyclictest2821471-21turbostat20:55:010
2688381991614,1cyclictest2821471-21turbostat20:55:000
2688381991613,2cyclictest0-21swapper/023:20:000
2688381991612,2cyclictest0-21swapper/022:35:000
2688381991611,3cyclictest3072303-21gltestperf00:15:170
2688387991512,2cyclictest0-21swapper/222:35:002
2688387991512,2cyclictest0-21swapper/200:15:002
268838799150,14cyclictest2952765-21chrt22:40:192
2688385991512,2cyclictest2943066-21kthreadcore22:30:241
2688385991512,2cyclictest0-21swapper/123:05:001
2688385991511,2cyclictest0-21swapper/122:00:011
2688385991511,2cyclictest0-21swapper/120:45:141
2688385991510,4cyclictest2777615-21idleruntime-cro20:20:001
268838599150,3cyclictest2923959-21cut22:15:211
268838599150,14cyclictest3015587-21grep23:30:151
268838199159,5cyclictest3614154-21systemd-journal21:20:000
268838199159,4cyclictest3614154-21systemd-journal21:07:070
268838199159,4cyclictest0-21swapper/019:55:000
268838199158,4cyclictest3614154-21systemd-journal22:20:000
2688381991513,1cyclictest2662768-21kworker/0:1+kacpid21:47:200
2688381991512,2cyclictest3053075-21latency_hist00:00:010
2688381991512,2cyclictest2771316-21cron20:15:000
2688381991511,3cyclictest2984127-21gltestperf23:05:170
2688381991511,3cyclictest2915528-21fschecks_time22:10:190
2688381991511,3cyclictest2871772-21cpuspeed_turbos21:35:150
2688381991511,3cyclictest2828537-21gltestperf21:00:160
2688381991511,3cyclictest171ktimers/019:20:120
2688381991511,2cyclictest0-21swapper/019:50:010
268791921512,2sleep30-21swapper/319:06:123
2688391991411,2cyclictest2796343-21gpgv20:35:003
2688391991410,3cyclictest491ktimers/320:25:133
268838799149,4cyclictest2909390-21gltestperf22:05:182
268838799149,4cyclictest2765006-21sshd20:08:582
268838799147,6cyclictest2821488-21idleruntime-cro20:55:002
268838799147,6cyclictest2821488-21idleruntime-cro20:55:002
2688387991411,2cyclictest3084896-21irqcore00:25:202
2688387991410,3cyclictest491-21avahi-daemon23:10:002
2688387991410,3cyclictest3034376-21gltestperf23:45:182
2688387991410,3cyclictest2964627-21cron22:50:002
2688387991410,3cyclictest2697360-21irqcore19:15:182
268838599143,3cyclictest753-21runrttasks23:15:201
2688385991412,1cyclictest3008587-21turbostat.cron23:25:001
2688385991412,1cyclictest2841369-21irqcore21:10:161
2688385991411,2cyclictest2853301-21fschecks_count21:20:151
2688385991411,2cyclictest2727312-21unixbench_multi19:35:241
2688385991411,2cyclictest0-21swapper/121:00:171
2688385991410,2cyclictest0-21swapper/121:40:001
268838599140,4cyclictest29619481kthreadcore22:45:211
268838599140,2cyclictest3062509-21tr00:05:191
268838199146,6cyclictest3614154-21systemd-journal20:25:000
2688381991412,1cyclictest2696121-21kthreadcore19:15:020
2688381991412,1cyclictest2662768-21kworker/0:1+kacpid19:25:230
2688381991412,1cyclictest2662768-21kworker/0:1+kacpid00:25:210
2688381991412,1cyclictest0-21swapper/023:00:000
2688381991411,2cyclictest502-21systemd-logind20:45:000
2688381991411,2cyclictest2870992-21unixbench_multi21:30:270
2688381991411,2cyclictest2662768-21kworker/0:1+kacpid22:22:370
2688381991411,2cyclictest2649381-21kworker/0:5+kacpid23:20:160
2688381991410,3cyclictest3614154-21systemd-journal21:45:000
2688381991410,3cyclictest2728181-21cpuspeed_turbos19:40:140
2688381991410,3cyclictest2662768-21kworker/0:1+kacpi_notify22:25:560
2688381991410,3cyclictest0-21swapper/019:10:160
268839199138,4cyclictest3001551-21ntpq23:15:223
268838799139,3cyclictest3058724-21grep00:00:252
268838799139,3cyclictest2859357-21cpuspeed_turbos21:25:152
268838799139,2cyclictest2834219-21munin-run21:05:002
268838799139,2cyclictest0-21swapper/221:20:182
268838799138,3cyclictest2752639-21sh20:00:002
268838799138,3cyclictest0-21swapper/223:45:012
2688387991311,1cyclictest0-21swapper/221:40:002
2688387991310,2cyclictest3079674-21tr00:20:172
2688387991310,2cyclictest2977379-21cpuspeed_turbos23:00:162
2688387991310,2cyclictest0-21swapper/221:45:122
2688387991310,2cyclictest0-21swapper/220:45:162
268838599139,3cyclictest506-21in:imuxsock19:50:011
268838599139,3cyclictest3097161-21cpuspeed_turbos00:35:151
268838599138,3cyclictest0-21swapper/120:50:251
268838599138,3cyclictest0-21swapper/120:50:251
2688385991310,2cyclictest2974083-21tr22:55:201
2688385991310,2cyclictest2934847-21irqrtprio22:25:201
2688385991310,2cyclictest2847158-21gltestperf21:15:201
2688385991310,2cyclictest0-21swapper/122:25:001
2688385991310,2cyclictest0-21swapper/120:25:001
2688385991310,2cyclictest0-21swapper/120:10:141
268838599130,2cyclictest30925771kthreadcore00:30:221
268838599130,1cyclictest0-21swapper/119:55:111
268838599130,12cyclictest2885136-21ps21:45:171
268838199139,3cyclictest3020864-21grep23:30:250
268838199139,3cyclictest2840972-21gltestperf21:10:150
268838199139,3cyclictest2827229-21unixbench_multi20:55:250
268838199139,3cyclictest2752671-21sendmail20:00:000
268838199139,3cyclictest2662768-21kworker/0:1+kacpi_notify20:30:140
268838199138,4cyclictest3614154-21systemd-journal00:05:580
268838199137,5cyclictest2760548-21kernelversion20:05:210
268838199137,4cyclictest3096797-21sh00:35:000
268838199136,5cyclictest3614154-21systemd-journal20:50:000
268838199136,5cyclictest3614154-21systemd-journal20:40:000
268838199135,6cyclictest493-21dbus-daemon22:05:000
2688381991311,2cyclictest2662768-21kworker/0:1+kacpid22:42:590
2688381991311,1cyclictest3027152-21kworker/0:0+kacpid23:45:210
2688381991311,1cyclictest2662768-21kworker/0:1+kacpid23:39:150
2688381991311,1cyclictest2662768-21kworker/0:1+kacpid22:53:240
2688381991311,1cyclictest2662768-21kworker/0:1+kacpid22:53:230
268838199131,11cyclictest0-21swapper/000:40:000
2688381991310,2cyclictest2662768-21kworker/0:1+kacpid23:10:200
2688381991310,2cyclictest2662768-21kworker/0:1+kacpid00:02:110
2688381991310,1cyclictest0-21swapper/020:00:200
268838199130,4cyclictest2727281-21unixbench-2d19:35:250
295798120,3rtkit-daemon0-21swapper/019:06:280
268839199129,2cyclictest0-21swapper/323:55:013
268839199127,4cyclictest3614154-21systemd-journal20:23:573
268839199127,4cyclictest2720882-21mailstats19:30:243
268838799129,2cyclictest2965188-21gltestperf22:50:272
268838799129,2cyclictest2965188-21gltestperf22:50:272
268838799129,2cyclictest2926944-21unixbench_multi22:15:272
268838799129,2cyclictest2841057-21gltestperf21:10:152
268838799128,3cyclictest2757769-21wc20:00:212
268838799126,4cyclictest493-21dbus-daemon19:15:002
268838799125,5cyclictest523-21gdbus23:30:002
268838599129,2cyclictest3083404-21systemctl00:20:221
268838599129,2cyclictest2826043-21kthreadcore20:55:201
268838599129,2cyclictest0-21swapper/121:05:251
268838599128,3cyclictest2797130-21gltestperf20:35:141
268838599126,4cyclictest2908037-21cat22:00:241
268838599121,3cyclictest2965010-21tr22:50:261
268838599121,3cyclictest2965010-21tr22:50:251
2688385991210,2cyclictest2688375-21cyclictest00:10:171
268838599120,1cyclictest0-21swapper/100:28:221
268838599120,11cyclictest2752891-21date20:00:011
268838599120,11cyclictest0-21swapper/123:11:031
268838199129,2cyclictest2649381-21kworker/0:5+kacpid19:30:040
268838199128,3cyclictest3008934-21cpuspeed_turbos23:25:170
268838199127,4cyclictest3041438-21sed23:50:190
268838199127,3cyclictest21750irq/159-ahci[0000:00:17.0]21:30:000
268838199126,4cyclictest733-21lldpd22:35:170
2688381991210,1cyclictest2662768-21kworker/0:1+kacpid22:46:160
2688381991210,1cyclictest2662768-21kworker/0:1+kacpid22:06:510
2688381991210,1cyclictest2662768-21kworker/0:1+kacpid20:25:260
2688381991210,1cyclictest2662768-21kworker/0:1+kacpid00:22:030
2688381991210,1cyclictest2649381-21kworker/0:5+kacpid23:00:230
268838199120,11cyclictest0-21swapper/021:58:260
26821582121,6sleep12681823-21apt-get19:05:021
2688391991110,1cyclictest0-21swapper/323:44:133
268838799115,4cyclictest3091837-21sed00:30:212
268838799115,4cyclictest3015412-21awk23:30:132
268838599118,2cyclictest6950irq/9-acpi20:42:381
268838599118,2cyclictest2878096-21cpuspeed_turbos21:40:141
268838599117,3cyclictest2989777-21systemctl23:10:001
268838599117,3cyclictest2864738-21unixbench_multi21:25:251
268838599117,2cyclictest0-21swapper/119:50:241
268838599116,4cyclictest2949052-21kthreadcore22:35:231
268838599116,3cyclictest2895438-21ntpq21:50:221
268838599110,1cyclictest0-21swapper/120:25:161
26879082118,2sleep20-21swapper/219:06:042
268839199107,2cyclictest3035357-21irqrtprio23:45:193
268839199106,3cyclictest2724604-21seq19:35:173
268839199105,4cyclictest2737497-21tr19:45:193
268839199105,4cyclictest2707482-21tr19:20:183
268838799107,2cyclictest401ktimers/222:20:152
268838799107,2cyclictest0-21swapper/223:50:192
268838799107,2cyclictest0-21swapper/222:25:182
268838799107,2cyclictest0-21swapper/221:05:152
268838799107,2cyclictest0-21swapper/220:55:142
268838799106,3cyclictest2972553-21cut22:55:192
268838799105,4cyclictest2950639-21tr22:35:242
268838799104,4cyclictest3065014-21/usr/sbin/munin00:05:262
268838599108,1cyclictest0-21swapper/120:05:211
268838599107,2cyclictest6950irq/9-acpi00:17:231
268838599105,3cyclictest0-21swapper/122:40:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional