You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 19:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 30, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
183397622850,9sleep24497-21PK-Backend07:05:212
197399990,8rtkit-daemon1972-21rtkit-daemon12:13:183
197399980,8rtkit-daemon1972-21rtkit-daemon12:22:103
197399970,8rtkit-daemon1972-21rtkit-daemon10:29:583
197399960,8rtkit-daemon1972-21rtkit-daemon10:12:583
197399940,9rtkit-daemon1972-21rtkit-daemon12:16:243
197399940,9rtkit-daemon1972-21rtkit-daemon09:38:323
197399920,9rtkit-daemon1972-21rtkit-daemon10:39:373
197399920,8rtkit-daemon1972-21rtkit-daemon09:57:163
18507192910,6sleep20-21swapper/207:50:282
197399880,9rtkit-daemon1972-21rtkit-daemon10:46:343
197399880,9rtkit-daemon1972-21rtkit-daemon10:02:023
197399880,8rtkit-daemon1972-21rtkit-daemon10:15:453
197399870,9rtkit-daemon1972-21rtkit-daemon11:09:583
197399850,8rtkit-daemon1972-21rtkit-daemon09:42:422
197399840,9rtkit-daemon1972-21rtkit-daemon10:56:013
197399840,8rtkit-daemon1972-21rtkit-daemon10:21:033
197399840,8rtkit-daemon1972-21rtkit-daemon09:43:473
19493502790,6sleep11949351-21find12:30:291
197399740,5rtkit-daemon0-21swapper/011:23:470
183375527341,25sleep30-21swapper/307:05:193
18654042700,7sleep14497-21PK-Backend08:35:021
183335427046,15sleep10-21swapper/107:05:131
18674252690,5sleep3411rcuc/308:40:153
18674252690,5sleep3411rcuc/308:40:153
183472326939,24sleep00-21swapper/007:09:020
197399670,9rtkit-daemon1972-21rtkit-daemon12:21:432
197399660,5rtkit-daemon0-21swapper/111:57:241
1834928996114,21cyclictest438-20systemd-journal08:03:302
183492899610,57cyclictest1972-21rtkit-daemon09:46:132
18699132600,5sleep30-21swapper/308:45:303
1834928995916,19cyclictest34-21ksoftirqd/209:10:002
1834928995916,17cyclictest34-21ksoftirqd/210:55:202
197399580,9rtkit-daemon1972-21rtkit-daemon12:39:382
197399580,9rtkit-daemon1972-21rtkit-daemon12:39:382
197399580,9rtkit-daemon1972-21rtkit-daemon10:37:232
197399580,9rtkit-daemon1972-21rtkit-daemon10:28:182
197399570,9rtkit-daemon1972-21rtkit-daemon10:09:223
197399570,9rtkit-daemon1972-21rtkit-daemon10:01:342
197399570,9rtkit-daemon1972-21rtkit-daemon09:00:321
1834928995630,8cyclictest34-21ksoftirqd/210:09:572
197399550,8rtkit-daemon1972-21rtkit-daemon10:44:272
197399550,8rtkit-daemon1972-21rtkit-daemon09:59:372
18583652550,13chrt261ksoftirqd/108:15:021
18472692550,5sleep10-21swapper/107:40:291
197399540,8rtkit-daemon1972-21rtkit-daemon10:24:552
183492499549,20cyclictest26-21ksoftirqd/112:10:021
197399530,8rtkit-daemon1972-21rtkit-daemon11:11:112
197399530,8rtkit-daemon1972-21rtkit-daemon09:35:532
197399530,43rtkit-daemon2078-21gmain07:35:220
19172942520,6chrt1916891-21/usr/sbin/munin11:00:170
19172942520,6chrt1916891-21/usr/sbin/munin11:00:160
1834928995212,20cyclictest34-21ksoftirqd/210:55:002
183493199501,31cyclictest4497-21PK-Backend11:50:233
1834928995018,17cyclictest34-21ksoftirqd/209:30:172
1834928995010,19cyclictest34-21ksoftirqd/209:15:302
1834931994921,24cyclictest42-21ksoftirqd/307:31:013
1834931994820,5cyclictest42-21ksoftirqd/311:11:313
183493199471,30cyclictest1932929-21sort11:45:193
183492899475,13cyclictest34-21ksoftirqd/212:12:022
183492899473,8cyclictest34-21ksoftirqd/211:34:592
183492899471,30cyclictest1836407-21chrt07:12:412
183492899471,14cyclictest34-21ksoftirqd/208:15:232
18518852460,14sleep1261ksoftirqd/107:55:141
183493199459,6cyclictest141rcu_preempt09:01:113
1834931994527,14cyclictest42-21ksoftirqd/308:04:303
183492899453,14cyclictest34-21ksoftirqd/211:00:152
183492899453,14cyclictest34-21ksoftirqd/211:00:152
1834928994516,5cyclictest141rcu_preempt12:30:202
1834928994512,9cyclictest34-21ksoftirqd/207:30:292
1834924994523,14cyclictest26-21ksoftirqd/112:05:021
183493199446,14cyclictest42-21ksoftirqd/308:55:123
1834931994418,8cyclictest42-21ksoftirqd/309:25:013
1834931994418,8cyclictest42-21ksoftirqd/308:09:543
183492899445,23cyclictest1842526-21apt-key07:30:022
1834924994422,13cyclictest26-21ksoftirqd/109:25:561
18965662430,5chrt0-21swapper/110:00:261
183493199438,6cyclictest141rcu_preempt09:45:243
1834931994323,17cyclictest1054-21snmpd08:27:273
1834931994311,16cyclictest42-21ksoftirqd/309:10:243
183492899435,17cyclictest1935127-21sshd11:50:292
183492899432,20cyclictest34-21ksoftirqd/210:15:022
1834924994326,12cyclictest438-20systemd-journal11:55:001
1834924994322,13cyclictest26-21ksoftirqd/108:40:291
1834924994322,13cyclictest26-21ksoftirqd/108:40:291
1834931994225,13cyclictest42-21ksoftirqd/307:16:093
1834931994224,13cyclictest42-21ksoftirqd/311:00:093
1834931994224,13cyclictest42-21ksoftirqd/311:00:083
1834931994222,12cyclictest42-21ksoftirqd/311:27:283
1834931994222,12cyclictest42-21ksoftirqd/311:27:273
1834931994215,9cyclictest42-21ksoftirqd/312:25:593
1834931994211,15cyclictest42-21ksoftirqd/308:30:203
183492899426,19cyclictest34-21ksoftirqd/209:50:232
183492899426,19cyclictest34-21ksoftirqd/209:50:222
183492899426,18cyclictest34-21ksoftirqd/209:26:082
1834928994221,8cyclictest34-21ksoftirqd/207:16:492
183492899421,22cyclictest438-20systemd-journal10:32:072
183492899421,22cyclictest1864490-21chrt08:30:242
1834928994211,17cyclictest34-21ksoftirqd/211:22:452
1834924994224,14cyclictest26-21ksoftirqd/111:05:201
1834924994223,12cyclictest26-21ksoftirqd/107:10:211
1834924994221,7cyclictest26-21ksoftirqd/110:30:241
18726992410,5sleep24497-21PK-Backend08:55:112
1834931994124,13cyclictest42-21ksoftirqd/308:14:103
1834931994123,14cyclictest42-21ksoftirqd/310:30:293
1834931994123,13cyclictest42-21ksoftirqd/311:55:123
1834931994111,17cyclictest42-21ksoftirqd/311:15:323
183492899419,15cyclictest1866922-21cron08:40:002
183492899415,5cyclictest141rcu_preempt09:25:022
183492899415,14cyclictest34-21ksoftirqd/211:09:282
183492899413,12cyclictest34-21ksoftirqd/210:15:212
1834928994129,9cyclictest34-21ksoftirqd/208:21:402
1834928994120,8cyclictest34-21ksoftirqd/207:20:172
183492899411,19cyclictest34-21ksoftirqd/212:25:022
1834924994127,11cyclictest26-21ksoftirqd/107:20:261
183492499412,35cyclictest1858978-21irqstats08:15:211
1834924994123,14cyclictest26-21ksoftirqd/112:10:311
1834924994123,14cyclictest26-21ksoftirqd/111:15:151
1834924994123,14cyclictest26-21ksoftirqd/111:10:181
1834924994123,13cyclictest1915730-21memory10:55:231
1834924994121,16cyclictest26-21ksoftirqd/112:25:011
1834924994120,13cyclictest26-21ksoftirqd/108:55:281
1834924994120,13cyclictest26-21ksoftirqd/108:25:241
1834924994119,14cyclictest26-21ksoftirqd/108:50:011
1834924994119,13cyclictest26-21ksoftirqd/107:35:271
183492499411,5cyclictest141rcu_preempt08:05:011
1834921994124,12cyclictest438-20systemd-journal12:17:000
183493199409,15cyclictest42-21ksoftirqd/309:05:143
1834931994023,13cyclictest42-21ksoftirqd/310:51:313
1834931994023,13cyclictest42-21ksoftirqd/309:51:433
1834931994023,13cyclictest42-21ksoftirqd/309:51:433
1834931994023,13cyclictest42-21ksoftirqd/309:28:253
1834931994023,13cyclictest42-21ksoftirqd/308:23:173
1834931994023,13cyclictest42-21ksoftirqd/308:16:133
1834931994023,13cyclictest42-21ksoftirqd/307:40:583
1834931994022,14cyclictest42-21ksoftirqd/311:40:053
1834931994022,14cyclictest42-21ksoftirqd/308:39:303
1834931994022,14cyclictest42-21ksoftirqd/307:39:133
1834931994022,14cyclictest42-21ksoftirqd/307:20:423
1834931994021,15cyclictest42-21ksoftirqd/312:32:093
1834931994021,15cyclictest42-21ksoftirqd/307:25:093
1834931994012,16cyclictest42-21ksoftirqd/307:46:123
1834931994012,13cyclictest1880127-21cut09:15:213
1834931994011,15cyclictest42-21ksoftirqd/312:37:123
1834931994011,15cyclictest42-21ksoftirqd/312:37:113
183492899402,9cyclictest0-21swapper/207:35:122
183492899402,21cyclictest34-21ksoftirqd/211:25:222
183492899402,21cyclictest34-21ksoftirqd/211:25:222
183492899402,20cyclictest34-21ksoftirqd/207:45:122
1834928994021,8cyclictest34-21ksoftirqd/212:05:002
1834928994020,8cyclictest34-21ksoftirqd/209:10:282
1834928994016,9cyclictest34-21ksoftirqd/211:55:302
183492499407,10cyclictest26-21ksoftirqd/108:10:001
1834924994023,6cyclictest26-21ksoftirqd/109:35:371
1834924994023,13cyclictest26-21ksoftirqd/109:10:011
1834924994023,13cyclictest26-21ksoftirqd/107:32:301
1834924994022,6cyclictest26-21ksoftirqd/110:40:201
1834924994022,6cyclictest26-21ksoftirqd/109:45:451
1834924994022,14cyclictest26-21ksoftirqd/111:40:011
1834924994022,14cyclictest26-21ksoftirqd/109:40:241
1834924994022,14cyclictest26-21ksoftirqd/109:35:011
1834924994021,12cyclictest26-21ksoftirqd/109:15:241
1834924994018,13cyclictest26-21ksoftirqd/109:55:281
1834921994013,7cyclictest141rcu_preempt11:50:200
1834931993922,13cyclictest42-21ksoftirqd/312:05:563
1834931993922,13cyclictest42-21ksoftirqd/312:00:443
1834931993922,13cyclictest42-21ksoftirqd/311:38:403
1834931993922,13cyclictest42-21ksoftirqd/310:40:563
1834931993922,13cyclictest42-21ksoftirqd/308:52:313
1834931993922,13cyclictest42-21ksoftirqd/307:55:553
1834931993922,13cyclictest1054-21snmpd11:33:053
1834931993921,14cyclictest42-21ksoftirqd/311:20:163
1834931993921,14cyclictest42-21ksoftirqd/307:51:293
1834931993911,16cyclictest1885513-21taskset09:30:203
183492899397,8cyclictest34-21ksoftirqd/208:40:232
183492899397,8cyclictest34-21ksoftirqd/208:40:222
183492899397,7cyclictest34-21ksoftirqd/208:46:572
183492899397,12cyclictest34-21ksoftirqd/210:45:292
183492899393,19cyclictest34-21ksoftirqd/208:25:142
1834928993918,8cyclictest34-21ksoftirqd/211:40:452
183492899391,21cyclictest1939570-21cut12:05:162
183492899391,17cyclictest34-21ksoftirqd/208:50:082
183492899391,14cyclictest956-21systemd-logind12:19:492
183492899391,14cyclictest34-21ksoftirqd/208:05:332
183492499397,15cyclictest26-21ksoftirqd/111:35:001
1834924993929,6cyclictest26-21ksoftirqd/112:25:191
1834924993924,11cyclictest1898981-21cron10:10:001
1834924993923,13cyclictest26-21ksoftirqd/112:35:141
1834924993923,13cyclictest26-21ksoftirqd/112:35:141
1834924993923,12cyclictest26-21ksoftirqd/108:50:201
1834924993922,5cyclictest26-21ksoftirqd/109:50:151
1834924993922,5cyclictest26-21ksoftirqd/109:50:151
1834924993922,13cyclictest26-21ksoftirqd/111:40:131
1834924993922,13cyclictest26-21ksoftirqd/111:20:151
1834924993922,13cyclictest26-21ksoftirqd/109:15:021
1834924993921,14cyclictest26-21ksoftirqd/111:03:381
1834924993921,14cyclictest26-21ksoftirqd/111:03:381
1834924993921,13cyclictest26-21ksoftirqd/110:15:271
1834924993921,13cyclictest26-21ksoftirqd/108:21:391
1834924993920,15cyclictest26-21ksoftirqd/110:45:531
1834924993919,13cyclictest26-21ksoftirqd/109:22:541
1834924993919,12cyclictest26-21ksoftirqd/110:25:281
183492499391,15cyclictest1904188-21cron10:25:001
1834921993922,13cyclictest1942665-21unin-run12:15:010
1834921993922,12cyclictest438-20systemd-journal09:40:000
1834921993921,13cyclictest438-20systemd-journal12:05:000
1834931993821,13cyclictest42-21ksoftirqd/307:10:323
183492899389,5cyclictest141rcu_preempt11:45:262
183492899382,16cyclictest0-21swapper/208:12:362
183492899382,14cyclictest0-21swapper/207:40:532
183492899381,18cyclictest34-21ksoftirqd/211:18:292
1834924993822,5cyclictest26-21ksoftirqd/111:45:271
1834924993822,5cyclictest26-21ksoftirqd/111:25:171
1834924993822,5cyclictest26-21ksoftirqd/111:25:171
1834924993822,13cyclictest26-21ksoftirqd/108:35:231
1834924993822,13cyclictest26-21ksoftirqd/107:15:031
1834924993822,12cyclictest26-21ksoftirqd/112:15:301
1834924993821,13cyclictest26-21ksoftirqd/110:50:421
1834924993821,13cyclictest26-21ksoftirqd/107:51:381
1834924993821,13cyclictest26-21ksoftirqd/107:45:281
1834924993820,6cyclictest26-21ksoftirqd/110:35:221
1834924993820,14cyclictest26-21ksoftirqd/110:10:281
1834924993820,14cyclictest26-21ksoftirqd/107:25:241
183492199387,6cyclictest19493532awk12:30:290
1834921993829,6cyclictest1872203-40apt-check08:52:560
1834921993824,10cyclictest1842290-21turbostat07:30:010
1834921993822,11cyclictest438-20systemd-journal08:21:390
1834921993814,20cyclictest1872698-21/usr/sbin/munin08:55:110
1834928993716,8cyclictest34-21ksoftirqd/207:56:392
1834928993713,6cyclictest141rcu_preempt11:36:242
183492899371,14cyclictest34-21ksoftirqd/209:05:012
1834921993725,8cyclictest1921840-21turbostat11:15:000
1834921993723,10cyclictest13-21ksoftirqd/011:55:300
1834921993720,13cyclictest1878351-21if_err_enp2s009:10:200
1834921993720,13cyclictest1869683-21ntp_states08:45:270
1834921993720,13cyclictest1836217-21uptime07:10:300
1834921993720,12cyclictest1904197-21ls10:25:000
1834921993719,13cyclictest1894244-21expr09:55:140
183492199371,6cyclictest1859973-21basename08:20:010
183492199371,16cyclictest1867118-21cat08:40:020
197399360,5rtkit-daemon0-21swapper/011:29:430
197399360,5rtkit-daemon0-21swapper/011:29:420
1834921993625,7cyclictest1885770-21cut09:30:240
1834921993621,11cyclictest1898982-21cron10:09:590
183492199361,9cyclictest0-21swapper/012:37:580
183492199361,9cyclictest0-21swapper/012:37:570
1834921993619,12cyclictest1940306-21sensors_temp12:05:290
1834921993617,15cyclictest1847107-21awk07:40:240
183492199361,30cyclictest438-20systemd-journal07:16:300
1834921993522,9cyclictest2069-21gnome-shell08:32:090
1834921993520,11cyclictest1930456-21sshd11:39:290
1834921993520,11cyclictest1904848-21sshd10:25:200
1834921993519,12cyclictest1933760-21sshd11:47:280
1834921993519,12cyclictest1853068-21pluginstate08:00:010
1834921993518,12cyclictest1913989-21cat10:50:220
1834921993518,11cyclictest438-20systemd-journal07:20:020
183492199351,6cyclictest141rcu_preempt11:43:300
1834921993511,6cyclictest141rcu_preempt09:15:130
183492199351,10cyclictest0-21swapper/009:04:080
197399340,27rtkit-daemon1941-21systemd07:51:460
1834921993419,11cyclictest1858215-21cron08:15:000
1834921993418,12cyclictest1849039-21grep07:45:310
1834921993418,11cyclictest438-20systemd-journal09:40:310
1834921993417,13cyclictest1915498-21cat10:55:160
183492199341,30cyclictest0-21swapper/007:32:060
1834921993410,6cyclictest141rcu_preempt09:25:010
1834921993322,8cyclictest1911001-21timerwakeupswit10:40:310
1834921993322,6cyclictest13-21ksoftirqd/011:18:570
1834921993320,9cyclictest1907656-40apt-check10:32:030
1834921993319,10cyclictest1948145-21/usr/sbin/munin12:30:010
1834921993319,10cyclictest1884706-21sshd09:28:300
183492199331,8cyclictest0-21swapper/011:05:210
1834921993317,12cyclictest1897273-21munin-run10:05:000
1834921993317,12cyclictest1892440-21sort09:50:150
1834921993317,12cyclictest1892440-21sort09:50:150
1834921993313,16cyclictest1867177-21/usr/sbin/munin08:40:300
1834921993313,16cyclictest1867177-21/usr/sbin/munin08:40:300
183492199329,5cyclictest141rcu_preempt10:10:290
1834921993226,4cyclictest0-21swapper/011:31:390
1834921993217,11cyclictest1854317-21cat08:00:300
1834921993216,11cyclictest438-20systemd-journal08:30:000
1834921993216,11cyclictest1908375-21cut10:35:140
1834921993215,12cyclictest1855038-21awk08:05:150
183492199321,4cyclictest0-21swapper/010:18:100
183492199321,29cyclictest0-21swapper/012:23:150
183492199319,18cyclictest1876528-21df_inode09:05:160
1834921993116,11cyclictest1891106-21sed09:45:200
183492199311,27cyclictest0-21swapper/010:47:520
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional