You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 07:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Thu Dec 04, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1843061994841,4cyclictest1936472-21kworker/u8:2+events_unbound23:58:302
1843065994631,11cyclictest391ktimers/320:20:153
1843052994536,5cyclictest1881835-21cron21:00:010
1843052994533,9cyclictest121ktimers/020:05:150
1843061994435,5cyclictest1919607-21idleruntime-cro22:50:002
1843052994431,10cyclictest121ktimers/020:20:160
1843061994334,6cyclictest1902551-21sendmail22:00:012
1843061994232,6cyclictest0-21swapper/221:40:002
1843061994129,7cyclictest0-21swapper/220:40:282
1843056994135,4cyclictest1938589-21cron23:45:011
1843052994131,8cyclictest121ktimers/021:40:140
1843052994131,8cyclictest121ktimers/021:40:130
1843061994031,5cyclictest1863075-21seq20:05:192
1843056994030,6cyclictest0-21swapper/121:21:341
1843052994031,6cyclictest1932832-21unixbench_singl23:25:250
1843056993931,4cyclictest0-21swapper/122:00:011
1843056993928,7cyclictest1858811-21grep19:50:261
1843056993927,8cyclictest1919600-21idleruntime-cro22:50:001
1843052993930,6cyclictest1891688-21grep21:25:280
1843065993829,6cyclictest1886967-21sh21:15:003
1843061993826,9cyclictest311ktimers/223:30:122
1843061993826,9cyclictest311ktimers/223:30:112
1843061993825,9cyclictest311ktimers/220:15:122
1843056993831,4cyclictest1916190-21sendmail22:40:001
1843056993830,4cyclictest0-21swapper/120:20:011
1843061993730,4cyclictest1936651-21cron23:40:002
1843061993726,7cyclictest0-21swapper/219:20:012
1843061993725,4cyclictest131rcu_preempt19:52:592
1843056993728,5cyclictest1850279-21grep19:25:251
1843056993710,9cyclictest1904427-21date22:05:011
1843065993629,3cyclictest0-21swapper/322:40:013
1843061993629,3cyclictest0-21swapper/220:35:002
1843061993626,6cyclictest0-21swapper/220:45:252
1843061993623,5cyclictest1947034-21kworker/u8:2+events_unbound00:19:422
1843061993623,5cyclictest131rcu_preempt22:10:012
1843061993623,5cyclictest131rcu_preempt22:10:012
1843061993618,10cyclictest131rcu_preempt21:25:192
1843061993616,3cyclictest131rcu_preempt00:25:182
1843061993616,3cyclictest131rcu_preempt00:25:182
1843056993627,6cyclictest1952630-21cut00:25:171
1843056993627,6cyclictest1952630-21cut00:25:171
1843052993627,6cyclictest1940659-21fschecks_count23:50:160
1843065993525,6cyclictest1909351-21cron22:20:003
1843061993525,6cyclictest0-21swapper/223:40:282
1843061993525,6cyclictest0-21swapper/219:35:272
1843061993521,5cyclictest131rcu_preempt21:20:012
1843061993521,10cyclictest311ktimers/222:10:152
1843061993519,4cyclictest131rcu_preempt19:58:082
1843061993511,13cyclictest131rcu_preempt23:20:132
1843056993529,3cyclictest0-21swapper/119:35:001
1843056993528,3cyclictest0-21swapper/122:05:191
1843056993528,3cyclictest0-21swapper/122:05:191
1843056993527,4cyclictest0-21swapper/120:25:151
1843056993526,5cyclictest1891691-21unixbench_singl21:25:281
1843052993526,3cyclictest0-21swapper/019:15:030
1843052993510,6cyclictest11-21ksoftirqd/021:50:270
1843065993427,4cyclictest1867982-21cron20:20:013
1843065993427,4cyclictest0-21swapper/319:20:013
1843065993425,5cyclictest0-21swapper/321:35:003
184306199348,8cyclictest32-21ksoftirqd/221:55:022
1843061993425,4cyclictest0-21swapper/200:15:002
1843061993424,6cyclictest0-21swapper/223:10:252
1843061993423,7cyclictest0-21swapper/221:20:272
1843061993423,6cyclictest0-21swapper/221:40:252
1843061993423,6cyclictest0-21swapper/221:40:252
1843061993422,7cyclictest0-21swapper/223:05:262
1843061993422,5cyclictest131rcu_preempt20:25:152
1843061993419,9cyclictest131rcu_preempt19:43:422
1843061993416,10cyclictest131rcu_preempt00:20:162
1843056993428,3cyclictest1909612-21apt-get22:20:131
1843056993424,7cyclictest1945404-21auditctl00:05:001
1843052993425,6cyclictest0-21swapper/020:55:010
1843052993425,5cyclictest1875021-21sendmail20:40:010
1843052993422,8cyclictest1870989-21grep20:25:240
1843052993416,7cyclictest131rcu_preempt00:02:470
1843052993410,6cyclictest11-21ksoftirqd/022:35:010
1843065993326,4cyclictest0-21swapper/322:55:003
1843065993325,4cyclictest0-21swapper/300:10:273
1843065993322,7cyclictest0-21swapper/321:30:013
184306199338,21cyclictest311ktimers/220:40:012
1843061993325,4cyclictest0-21swapper/222:05:012
184306199332,15cyclictest32-21ksoftirqd/222:15:282
1843061993320,5cyclictest131rcu_preempt19:45:172
1843061993320,5cyclictest131rcu_preempt19:45:172
1843061993315,3cyclictest0-21swapper/220:01:192
1843056993325,4cyclictest0-21swapper/121:15:171
184305299338,13cyclictest0-21swapper/021:00:150
184305299335,8cyclictest1849097-21latency_hist19:25:010
1843052993310,6cyclictest11-21ksoftirqd/020:05:030
1843052993310,5cyclictest11-21ksoftirqd/023:10:250
1843052993310,4cyclictest11-21ksoftirqd/021:10:260
1843065993225,4cyclictest1940241-21kworker/u8:0+events_unbound00:23:563
1843065993224,6cyclictest165-21systemd-journal19:29:523
1843065993224,4cyclictest0-21swapper/323:10:163
1843065993223,5cyclictest0-21swapper/321:20:263
184306199326,12cyclictest311ktimers/222:28:552
1843061993223,6cyclictest0-21swapper/222:45:012
1843061993222,7cyclictest301rcuc/220:57:552
1843061993222,6cyclictest1844788-21cut19:15:012
1843061993220,5cyclictest131rcu_preempt00:00:192
1843061993218,6cyclictest1914357-21kworker/u8:1+events_unbound22:50:352
1843061993217,9cyclictest131rcu_preempt20:20:312
1843061993215,5cyclictest131rcu_preempt23:53:302
1843061993215,10cyclictest131rcu_preempt19:21:102
1843061993215,10cyclictest131rcu_preempt00:30:242
1843061993214,10cyclictest131rcu_preempt19:25:132
1843061993212,8cyclictest1947237-21latency_hist00:10:002
1843061993211,5cyclictest131rcu_preempt20:10:152
1843056993225,4cyclictest0-21swapper/100:05:291
1843056993224,5cyclictest0-21swapper/100:20:001
1843056993224,4cyclictest0-21swapper/119:45:251
1843056993224,4cyclictest0-21swapper/119:45:251
1843056993219,10cyclictest1843662-21/usr/sbin/munin19:10:191
1843056993217,12cyclictest1860852-21cron19:59:001
184305299324,7cyclictest131rcu_preempt22:00:490
1843052993225,3cyclictest0-21swapper/023:20:000
1843052993222,7cyclictest1917430-21grep22:40:240
1843052993222,6cyclictest0-21swapper/023:00:010
1843052993210,5cyclictest11-21ksoftirqd/023:30:170
1843052993210,5cyclictest11-21ksoftirqd/023:30:170
1843065993123,5cyclictest1888688-21idleruntime-cro21:20:003
1843065993123,4cyclictest0-21swapper/323:35:283
1843065993122,5cyclictest1848575-21grep19:20:263
1843065993122,5cyclictest0-21swapper/322:10:283
1843065993116,10cyclictest1-21systemd22:31:473
1843061993125,3cyclictest301rcuc/219:30:012
1843061993123,4cyclictest0-21swapper/223:15:252
1843061993121,4cyclictest131rcu_preempt21:00:202
1843061993118,7cyclictest1912506-21kworker/u8:0+events_unbound22:32:222
1843061993117,8cyclictest1914357-21kworker/u8:1+events_unbound22:37:552
1843061993117,7cyclictest1881514-21kworker/u8:3+events_unbound21:07:592
1843061993114,9cyclictest131rcu_preempt20:50:242
1843061993114,4cyclictest131rcu_preempt21:10:152
1843061993114,10cyclictest131rcu_preempt22:55:192
1843061993113,4cyclictest131rcu_preempt23:48:562
1843061993111,12cyclictest131rcu_preempt22:20:222
1843056993122,6cyclictest53-21kcompactd019:35:271
1843056993117,11cyclictest1930535-21memory23:20:211
1843056993112,13cyclictest131rcu_preempt20:10:171
184305299319,4cyclictest11-21ksoftirqd/021:20:190
184305299316,7cyclictest11-21ksoftirqd/020:40:170
1843065993023,4cyclictest0-21swapper/323:05:263
1843065993022,5cyclictest0-21swapper/300:40:003
1843065993022,4cyclictest0-21swapper/323:45:243
1843065993022,4cyclictest0-21swapper/323:01:533
1843065993022,4cyclictest0-21swapper/320:56:283
1843065993020,7cyclictest0-21swapper/300:00:003
1843065993020,6cyclictest0-21swapper/320:50:263
1843065993018,8cyclictest1-21systemd00:28:563
1843065993018,8cyclictest1-21systemd00:28:563
184306199309,7cyclictest32-21ksoftirqd/200:40:002
1843061993018,3cyclictest131rcu_preempt21:45:232
1843061993012,5cyclictest131rcu_preempt23:00:482
1843061993012,5cyclictest131rcu_preempt21:30:162
184305699309,5cyclictest24-21ksoftirqd/119:18:271
1843056993024,3cyclictest1940520-21apt-get23:50:131
1843056993024,3cyclictest1923254-21apt-get23:00:131
1843056993024,3cyclictest1921535-21apt-get22:55:131
1843056993024,3cyclictest1919830-21apt-get22:50:121
1843056993022,4cyclictest0-21swapper/100:20:261
1843056993020,6cyclictest1927640-21grep23:10:251
1843056993020,6cyclictest0-21swapper/123:15:251
1843056993017,11cyclictest1900090-21/usr/sbin/munin21:50:271
1843056993017,10cyclictest1948714-21cron00:14:001
1843056993016,11cyclictest1914085-21/usr/sbin/munin22:30:261
184305299309,5cyclictest11-21ksoftirqd/023:58:420
184305299309,10cyclictest1950512-21latency_hist00:20:000
184305299308,6cyclictest0-21swapper/023:45:130
184305299308,15cyclictest131rcu_preempt00:08:370
184305299307,4cyclictest0-21swapper/023:35:170
1843052993019,7cyclictest1908893-21grep22:15:280
1843052993015,7cyclictest0-21swapper/000:23:470
1843052993010,6cyclictest11-21ksoftirqd/019:50:140
1843065992922,4cyclictest0-21swapper/323:20:013
1843065992921,4cyclictest0-21swapper/323:30:243
1843065992921,4cyclictest0-21swapper/323:30:243
1843065992921,4cyclictest0-21swapper/321:40:003
1843065992921,4cyclictest0-21swapper/320:10:013
1843065992920,6cyclictest1861928-21mailstats20:00:253
1843056992924,3cyclictest1878671-21apt-get20:50:141
1843056992922,4cyclictest1875268-21apt-get20:40:141
1843056992919,7cyclictest1934863-21cron23:34:001
1843056992919,7cyclictest1934863-21cron23:34:001
1843056992919,7cyclictest1932624-21/usr/sbin/munin23:25:231
1843056992919,7cyclictest1916426-21/usr/sbin/munin22:40:251
1843056992918,8cyclictest1897791-21/usr/sbin/munin21:45:191
1843056992918,8cyclictest1883306-21cron21:02:001
1843056992918,2cyclictest131rcu_preempt00:36:371
1843056992917,9cyclictest1906241-21/usr/sbin/munin22:10:161
1843056992917,7cyclictest1873763-21cut20:35:191
1843056992914,12cyclictest1954282-21diskstats00:30:151
1843056992914,11cyclictest1893325-21grep21:30:261
1843056992914,11cyclictest1863884-21tail20:05:281
184305299298,6cyclictest0-21swapper/022:05:270
184305299298,6cyclictest0-21swapper/022:05:260
1843052992921,4cyclictest0-21swapper/000:40:010
1843052992919,6cyclictest1907218-21grep22:10:280
1843065992821,5cyclictest1854580-21kworker/u8:1+events_unbound19:46:153
1843065992821,5cyclictest1854580-21kworker/u8:1+events_unbound19:46:153
1843065992821,4cyclictest1873114-21munin-plugin-st20:35:023
1843065992821,4cyclictest0-21swapper/319:10:263
1843065992820,4cyclictest0-21swapper/323:40:193
1843056992818,7cyclictest1925235-21/usr/sbin/munin23:05:171
1843056992818,7cyclictest1896119-21/usr/sbin/munin21:40:181
1843056992818,7cyclictest1896119-21/usr/sbin/munin21:40:181
1843056992817,8cyclictest1939946-21cron23:46:001
1843056992817,8cyclictest1935192-21apt-get23:35:141
1843056992817,8cyclictest1894475-21meminfo21:35:201
1843056992817,8cyclictest1884246-21memory21:05:211
1843056992817,8cyclictest1880775-21/usr/sbin/munin20:55:201
1843056992817,8cyclictest1871752-21diskstats20:30:151
184305299289,5cyclictest0-21swapper/022:25:200
184305299289,5cyclictest0-21swapper/019:40:190
184305299287,6cyclictest0-21swapper/000:13:120
184305299286,6cyclictest0-21swapper/019:15:200
1843052992811,2cyclictest0-21swapper/023:25:010
184305299280,16cyclictest0-21swapper/023:07:460
1843065992719,6cyclictest1912506-21kworker/u8:0+events_unbound22:46:513
1843065992719,5cyclictest1875796-21kworker/u8:2+events_unbound20:41:273
1843065992719,4cyclictest1905546-21head22:05:293
1843065992719,4cyclictest1905546-21head22:05:283
1843056992717,8cyclictest1942220-21apt-get23:55:121
1843056992717,7cyclictest1886107-21/usr/sbin/munin21:10:241
1843056992717,7cyclictest0-21swapper/120:00:171
1843056992716,8cyclictest1848644-21cron19:21:011
1843056992715,9cyclictest1855831-21cron19:45:001
1843056992711,12cyclictest440-21cron20:50:011
184305299279,9cyclictest131rcu_preempt22:50:560
184305299276,12cyclictest0-21swapper/021:33:510
184305299273,9cyclictest1885230-21taskset21:09:410
1843052992720,4cyclictest1954306-21fschecks_count00:30:160
1843052992720,3cyclictest0-21swapper/021:50:000
1843052992719,4cyclictest0-21swapper/019:55:270
1843052992718,5cyclictest0-21swapper/019:40:010
184306599269,4cyclictest0-21swapper/321:05:123
1843065992615,7cyclictest1-21systemd19:36:133
1843056992617,6cyclictest1911418-21df22:25:151
184305299269,4cyclictest131rcu_preempt22:20:160
184305299268,5cyclictest0-21swapper/023:00:160
184305299268,3cyclictest131rcu_preempt19:31:090
184305299266,11cyclictest0-21swapper/020:30:260
184305299261,9cyclictest18671392/usr/sbin/munin20:15:220
1843052992618,5cyclictest0-21swapper/023:40:280
1843052992618,4cyclictest1915706-21unixbench_multi22:35:260
1843052992615,6cyclictest0-21swapper/019:25:240
1843052992611,12cyclictest0-21swapper/020:10:140
1843052992610,7cyclictest131rcu_preempt21:16:510
1843065992517,4cyclictest0-21swapper/320:45:243
1843065992517,4cyclictest0-21swapper/300:35:003
1843065992515,6cyclictest1950539-21grep00:20:013
184306599250,13cyclictest0-21swapper/319:31:123
184305299253,10cyclictest11-21ksoftirqd/020:46:380
1843052992518,3cyclictest0-21swapper/022:00:010
1843052992510,8cyclictest131rcu_preempt19:45:240
1843052992510,8cyclictest131rcu_preempt19:45:240
1843052992510,8cyclictest131rcu_preempt00:29:000
1843052992510,8cyclictest131rcu_preempt00:29:000
184306599247,4cyclictest131rcu_preempt22:55:263
1843065992415,6cyclictest1865494-21fgrep20:10:233
1843065992415,5cyclictest0-21swapper/320:25:233
1843065992413,7cyclictest0-21swapper/322:45:003
1843052992415,6cyclictest1918688-21ntpq22:45:200
1843065992316,4cyclictest0-21swapper/323:25:243
1843065992316,4cyclictest0-21swapper/319:55:223
1843065992216,3cyclictest1900333-21sshd21:51:393
1843065992213,6cyclictest1873546-21/usr/sbin/munin20:35:143
184306599221,18cyclictest1883128-21awk21:00:263
1843065992211,6cyclictest1941322-21wc23:50:253
1843065992113,6cyclictest391ktimers/323:20:203
1843065992113,6cyclictest1910027-21sed22:20:183
1843065992014,3cyclictest1947255-21latency_hist00:10:013
1843065992013,5cyclictest391ktimers/321:45:233
1843065992013,4cyclictest0-21swapper/319:40:253
1843065992013,3cyclictest0-21swapper/321:41:363
1843065992013,3cyclictest0-21swapper/321:41:363
184306599187,8cyclictest1858220-21memory19:50:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional