You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 22:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Aug 23, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
367764999894,14cyclictest3739701-21modprobe11:10:000
367764999877,74cyclictest0-21swapper/010:00:000
3677660998636,14cyclictest3708736-21ssh10:03:233
3677649998524,4cyclictest16850irq/134-800000009:35:010
367764999841,13cyclictest3711987-21cat10:10:020
3677660998330,17cyclictest3751034-21ssh11:34:123
3677649998328,14cyclictest3724169-21ssh10:36:240
3677649998327,11cyclictest3718867-21ssh10:25:010
3677649998226,13cyclictest3778013-21ssh12:32:070
367764999812,76cyclictest0-21swapper/010:15:030
367764999812,76cyclictest0-21swapper/010:15:030
3677649998127,12cyclictest3735579-21ssh11:00:460
367764999811,37cyclictest3708871-21ssh10:03:400
3677649998026,46cyclictest275-21dbus-daemon07:15:000
3677649997932,41cyclictest309-21in:imuxsock12:20:010
3677660997834,17cyclictest3721514-21ssh10:30:403
3677649997827,12cyclictest3734865-21ssh10:59:380
3677649997827,12cyclictest3734865-21ssh10:59:370
3677649997619,11cyclictest3719551-21ssh10:26:270
367764999753,8cyclictest3714033-21munin-run10:15:000
3677649997527,12cyclictest3755346-21ssh11:43:180
3677649997526,11cyclictest3766769-21ssh12:07:580
3677660997422,14cyclictest0-21swapper/308:16:123
367764999744,17cyclictest3681205-21cron08:19:590
3677649997428,4cyclictest16950irq/224-800000011:25:010
3677649997428,13cyclictest3757007-21ssh11:46:400
3677649997427,7cyclictest3698904-21ssh09:42:230
3677649997426,13cyclictest3769333-21ssh12:13:420
3677649997426,13cyclictest3769333-21ssh12:13:420
3677660997324,13cyclictest1-21systemd08:10:013
367764999733,8cyclictest3704711-21modprobe09:55:000
3677649997327,7cyclictest3763216-21ssh12:00:030
3677649997327,10cyclictest3780654-21ssh12:37:580
3677660997224,14cyclictest3746058-21ssh11:23:263
3677649997229,39cyclictest3774552-21cron12:25:000
3677649997228,4cyclictest131rcu_preempt07:50:020
3677649997218,12cyclictest3747826-21ssh11:27:180
3677649997218,12cyclictest3747826-21ssh11:27:180
367764999714,20cyclictest3678111-21grep07:20:010
3677649997127,13cyclictest3760064-21ssh11:53:330
3677649997117,14cyclictest3742556-21ssh11:15:410
367764999711,39cyclictest3702165-21ssh09:49:240
367764999711,39cyclictest3702165-21ssh09:49:230
3677649997111,17cyclictest16850irq/134-800000011:38:170
3677660997029,6cyclictest3738133-21ssh11:06:113
3677649997035,31cyclictest3688668-21ssh09:20:170
367764999702,8cyclictest3693099-21latency_hist09:30:020
367764999701,29cyclictest0-21swapper/009:05:000
3677660996956,10cyclictest3741357-21ssh11:13:273
3677660996924,14cyclictest3715118-21ssh10:17:053
3677660996924,14cyclictest3715118-21ssh10:17:043
367765699696,6cyclictest30-21ksoftirqd/208:00:012
367765699696,6cyclictest30-21ksoftirqd/208:00:002
3677656996927,28cyclictest3669271-21kworker/u8:207:25:012
367764999695,12cyclictest3751323-21modprobe11:35:000
3677649996922,13cyclictest3775542-21ssh12:26:510
367764999691,4cyclictest3730675-21sshd10:50:110
3677660996824,13cyclictest3731541-21ssh10:52:193
3677660996822,14cyclictest3779269-21ls12:35:023
3677649996818,17cyclictest16850irq/134-800000010:33:290
3677660996751,13cyclictest3780045-21ssh12:36:313
367764999673,24cyclictest3682044-21latency_hist08:35:010
3677649996729,7cyclictest12-21ksoftirqd/011:15:020
367764999672,5cyclictest0-21swapper/012:00:000
3677649996724,5cyclictest3730505-21latency_hist10:50:010
3677649996720,12cyclictest3682764-21sshd08:49:100
3677649996718,5cyclictest16850irq/134-800000009:39:200
3677660996654,9cyclictest3735539-21ssh11:00:413
3677660996652,11cyclictest3733884-21ssh10:57:213
3677660996652,11cyclictest3733884-21ssh10:57:213
3677660996620,14cyclictest3757630-21ssh11:48:093
3677660996546,15cyclictest3694196-21ssh09:32:123
3677660996443,17cyclictest3698949-21ssh09:42:283
3677649996425,8cyclictest12-21ksoftirqd/009:18:490
3677649996425,8cyclictest12-21ksoftirqd/009:18:490
3677660996350,10cyclictest3775552-21ssh12:26:533
3677660996346,13cyclictest3748921-21ssh11:29:473
3677660996346,13cyclictest3748921-21ssh11:29:473
3677660996344,15cyclictest3724325-21ssh10:36:433
367765699637,10cyclictest30-21ksoftirqd/212:40:022
367764999634,17cyclictest3686093-21cat09:15:020
3677660996250,9cyclictest3729145-21ssh10:47:033
3677660996247,12cyclictest3719116-21ssh10:25:213
3677660996246,13cyclictest3772469-21ssh12:20:043
3677660996245,13cyclictest3727119-21ssh10:42:533
3677660996245,13cyclictest3691558-21ssh09:26:263
3677660996223,13cyclictest364-21snmpd07:53:123
3677660996219,13cyclictest3683824-21diskmemload11:55:473
367765699627,12cyclictest30-21ksoftirqd/209:40:102
3677649996252,5cyclictest16950irq/224-800000007:39:590
367764999625,15cyclictest3682264-21cut08:40:010
3677660996148,10cyclictest3754874-21ssh11:42:103
3677660996148,10cyclictest3709869-21ssh10:05:293
3677660996147,10cyclictest3689647-21ssh09:22:323
3677660996145,13cyclictest3695952-21ssh09:35:503
3677660996145,12cyclictest3703060-21ssh09:50:583
3677660996144,13cyclictest3712179-21ssh10:10:313
3677660996144,13cyclictest3700798-21ssh09:46:123
3677660996144,13cyclictest3700798-21ssh09:46:123
3677660996143,15cyclictest3759195-21ssh11:51:193
3677660996141,16cyclictest364-21snmpd12:02:073
3677660996122,14cyclictest3681531-21cat08:25:023
3677649996126,7cyclictest12-21ksoftirqd/007:31:110
3677649996123,14cyclictest275-21dbus-daemon08:25:010
3677660996047,10cyclictest3751584-21ssh11:35:053
3677660996047,10cyclictest3686630-21ssh09:16:013
3677660996047,10cyclictest3686630-21ssh09:16:013
3677660996046,11cyclictest3705338-21ssh09:55:503
3677660996046,10cyclictest3683259-21sshd08:57:113
3677660996045,12cyclictest3765959-21ssh12:06:073
3677660996045,12cyclictest3716769-21ssh10:20:243
3677660996044,13cyclictest3768492-21ssh12:11:313
3677660996044,13cyclictest3768492-21ssh12:11:303
3677660996043,14cyclictest3743464-21ssh11:17:543
3677660996043,13cyclictest3680939-21sshd08:14:263
3677660996043,13cyclictest3680939-21sshd08:14:253
3677656996044,8cyclictest3686055-21kworker/2:110:54:292
3677654996014,8cyclictest23-21ksoftirqd/108:00:011
3677654996014,8cyclictest23-21ksoftirqd/108:00:001
367764999606,11cyclictest3679414-21munin-run07:45:010
367764999606,11cyclictest3679414-21munin-run07:45:010
367764999605,14cyclictest3683280-21modprobe08:59:590
367764999603,8cyclictest3684026-21latency_hist09:10:030
367764999603,8cyclictest3684026-21latency_hist09:10:030
3677649996020,10cyclictest3715530-21kworker/u8:110:44:190
3677660995946,10cyclictest3770406-21ssh12:15:433
3677660995946,10cyclictest3684491-21ssh09:11:203
3677660995943,11cyclictest3678142-21munin-run07:20:023
367765699595,7cyclictest30-21ksoftirqd/210:05:022
3677660995846,9cyclictest3683550-21sshd09:04:363
3677660995845,10cyclictest3678851-21sshd07:31:113
3677660995843,11cyclictest3680434-21sshd08:04:243
3677654995815,9cyclictest23-21ksoftirqd/110:20:011
3677654995815,9cyclictest23-21ksoftirqd/110:20:001
3677660995744,10cyclictest3682215-21sshd08:35:113
3677660995743,10cyclictest3683036-21munin-run08:55:013
367765699575,8cyclictest30-21ksoftirqd/211:10:012
367764999577,14cyclictest3682536-21cut08:45:000
3677660995649,4cyclictest3682014-21munin-run08:35:003
3677660995644,9cyclictest364-21snmpd07:48:293
3677660995643,10cyclictest364-21snmpd08:46:583
3677660995640,11cyclictest364-21snmpd08:43:413
367765699566,9cyclictest30-21ksoftirqd/209:26:332
3677656995610,14cyclictest30-21ksoftirqd/209:23:502
367765499566,8cyclictest23-21ksoftirqd/112:09:581
3677660995548,4cyclictest3681762-21munin-run08:30:023
3677660995546,6cyclictest3677853-21munin-run07:15:003
3677660995543,9cyclictest3679428-21munin-run07:45:013
3677660995543,9cyclictest3679428-21munin-run07:45:003
3677660995541,9cyclictest48-21kcompactd007:59:133
3677660995541,9cyclictest48-21kcompactd007:59:123
367765699557,11cyclictest30-21ksoftirqd/210:35:332
367765699556,8cyclictest30-21ksoftirqd/211:33:132
367765699556,14cyclictest30-21ksoftirqd/211:40:052
367765699555,9cyclictest30-21ksoftirqd/211:24:362
367765699555,7cyclictest30-21ksoftirqd/211:00:532
3677656995517,9cyclictest30-21ksoftirqd/207:35:012
3677656995512,9cyclictest30-21ksoftirqd/209:20:012
3677656995512,9cyclictest30-21ksoftirqd/209:20:002
367765499555,6cyclictest23-21ksoftirqd/112:35:011
3677660995440,11cyclictest3679291-21sed07:40:013
367765699547,10cyclictest30-21ksoftirqd/209:31:452
367765699545,9cyclictest30-21ksoftirqd/209:40:012
367765699544,9cyclictest30-21ksoftirqd/212:32:112
3677656995440,6cyclictest3686055-21kworker/2:110:49:022
367765499547,11cyclictest23-21ksoftirqd/110:24:411
367765499543,10cyclictest23-21ksoftirqd/109:31:581
367764999543,18cyclictest3680946-21modprobe08:15:010
367764999543,18cyclictest3680946-21modprobe08:15:010
3677660995345,4cyclictest3678643-21munin-run07:30:013
367765699536,9cyclictest30-21ksoftirqd/210:07:012
367765699536,10cyclictest30-21ksoftirqd/212:19:572
367765699535,10cyclictest30-21ksoftirqd/210:29:312
3677656995343,8cyclictest30-21ksoftirqd/209:10:112
3677656995311,12cyclictest30-21ksoftirqd/211:35:022
367765499539,8cyclictest23-21ksoftirqd/111:08:121
367765499535,8cyclictest23-21ksoftirqd/109:42:051
3677654995313,11cyclictest23-21ksoftirqd/109:20:001
3677654995313,11cyclictest23-21ksoftirqd/109:20:001
3677660995239,9cyclictest364-21snmpd09:06:173
3677660995239,9cyclictest364-21snmpd09:06:173
367765699527,9cyclictest30-21ksoftirqd/211:14:432
367765699525,9cyclictest30-21ksoftirqd/210:43:122
367765699525,9cyclictest30-21ksoftirqd/209:45:172
367765699525,9cyclictest30-21ksoftirqd/209:45:172
367765699524,8cyclictest30-21ksoftirqd/212:01:512
3677656995244,5cyclictest3671492-21kworker/2:310:21:092
3677656995212,8cyclictest30-21ksoftirqd/208:53:082
367765499526,9cyclictest23-21ksoftirqd/112:16:151
367765499525,8cyclictest23-21ksoftirqd/109:47:551
367765499525,8cyclictest23-21ksoftirqd/109:47:541
3677660995137,10cyclictest364-21snmpd07:20:353
367765699517,10cyclictest30-21ksoftirqd/208:50:012
367765699516,8cyclictest30-21ksoftirqd/209:57:212
367765699516,11cyclictest30-21ksoftirqd/212:28:382
367765699511,7cyclictest30-21ksoftirqd/212:05:402
367765499519,6cyclictest131rcu_preempt12:27:221
367765499516,6cyclictest23-21ksoftirqd/109:24:211
367765499514,8cyclictest23-21ksoftirqd/111:38:591
3677654995113,7cyclictest221rcuc/108:16:061
3677654995110,9cyclictest23-21ksoftirqd/111:04:591
3677649995127,4cyclictest3683045-21sh08:55:010
367765699507,12cyclictest30-21ksoftirqd/210:55:332
367765699507,12cyclictest30-21ksoftirqd/210:55:332
367765699506,8cyclictest30-21ksoftirqd/210:33:432
367765699505,9cyclictest30-21ksoftirqd/209:54:012
367765699505,7cyclictest30-21ksoftirqd/211:19:402
3677656995042,6cyclictest291rcuc/212:11:322
3677656995042,6cyclictest291rcuc/212:11:322
367765699504,10cyclictest30-21ksoftirqd/210:10:582
367765699503,8cyclictest30-21ksoftirqd/207:45:012
367765699503,8cyclictest30-21ksoftirqd/207:45:002
367765499508,8cyclictest23-21ksoftirqd/112:21:031
367765499507,6cyclictest23-21ksoftirqd/111:19:281
367765499507,6cyclictest23-21ksoftirqd/110:04:321
367765499506,5cyclictest23-21ksoftirqd/109:38:151
367765499505,6cyclictest23-21ksoftirqd/111:42:081
367765499505,6cyclictest23-21ksoftirqd/110:53:111
367765499505,6cyclictest23-21ksoftirqd/110:30:401
3677654995018,24cyclictest0-21swapper/110:28:581
3677654995010,5cyclictest131rcu_preempt11:56:451
3677649995032,13cyclictest275-21dbus-daemon08:00:010
3677649995032,13cyclictest275-21dbus-daemon08:00:000
3677649995026,16cyclictest275-21dbus-daemon07:30:010
367765699496,8cyclictest30-21ksoftirqd/210:15:492
367765699496,8cyclictest30-21ksoftirqd/210:15:482
367765699496,10cyclictest30-21ksoftirqd/212:24:122
3677656994940,6cyclictest3671492-21kworker/2:307:36:032
3677656994940,5cyclictest3664925-21kworker/2:108:00:332
367765699493,7cyclictest30-21ksoftirqd/211:46:292
367765499496,8cyclictest23-21ksoftirqd/111:32:191
367765499496,6cyclictest23-21ksoftirqd/111:27:571
367765499496,6cyclictest23-21ksoftirqd/111:27:571
367765499495,7cyclictest23-21ksoftirqd/112:39:541
367765499493,8cyclictest23-21ksoftirqd/111:46:171
3677654994933,13cyclictest3693255-21sed09:30:031
3677654994915,9cyclictest23-21ksoftirqd/110:41:191
367765699487,10cyclictest30-21ksoftirqd/211:52:062
3677656994841,4cyclictest3671492-21kworker/2:308:33:222
3677656994840,5cyclictest3730389-21kworker/2:211:59:322
3677656994840,5cyclictest3682799-21kworker/2:009:01:132
3677656994840,5cyclictest3682799-21kworker/2:008:55:212
3677656994840,5cyclictest3671492-21kworker/2:308:29:022
3677656994840,5cyclictest3671492-21kworker/2:308:24:402
3677656994840,5cyclictest3671492-21kworker/2:308:10:112
3677656994840,5cyclictest3671492-21kworker/2:308:10:112
3677656994840,5cyclictest3664925-21kworker/2:107:50:552
3677656994840,5cyclictest3664925-21kworker/2:107:18:332
367765499488,10cyclictest23-21ksoftirqd/107:35:021
367765499487,8cyclictest23-21ksoftirqd/111:50:371
367765499486,9cyclictest23-21ksoftirqd/112:11:381
367765499486,9cyclictest23-21ksoftirqd/112:11:381
367765499485,7cyclictest23-21ksoftirqd/109:57:461
3677654994840,5cyclictest23-21ksoftirqd/110:37:561
367765499483,14cyclictest23-21ksoftirqd/110:57:541
367765499483,14cyclictest23-21ksoftirqd/110:57:541
367765499481,10cyclictest23-21ksoftirqd/111:14:311
367765699476,11cyclictest30-21ksoftirqd/211:25:102
367765699476,11cyclictest30-21ksoftirqd/211:25:092
3677656994740,4cyclictest3682799-21kworker/2:009:05:132
3677656994740,4cyclictest3682799-21kworker/2:009:05:122
3677656994740,4cyclictest3664925-21kworker/2:107:27:372
3677656994739,5cyclictest3680730-21kworker/2:208:42:192
3677656994739,5cyclictest3671492-21kworker/2:308:38:382
3677656994739,5cyclictest3671492-21kworker/2:308:15:262
3677656994739,5cyclictest3664925-21kworker/2:108:06:552
3677656994739,5cyclictest3664925-21kworker/2:107:48:332
3677656994739,5cyclictest3664925-21kworker/2:107:12:002
367765499478,9cyclictest23-21ksoftirqd/109:51:471
367765499476,7cyclictest23-21ksoftirqd/112:03:031
367765499476,10cyclictest23-21ksoftirqd/108:40:001
367765499469,5cyclictest131rcu_preempt10:46:131
367765499467,23cyclictest3680147-21kworker/u8:108:30:031
367765499465,7cyclictest23-21ksoftirqd/109:11:371
367765499465,6cyclictest23-21ksoftirqd/110:14:111
367765499464,7cyclictest23-21ksoftirqd/108:15:021
367765499464,7cyclictest23-21ksoftirqd/108:15:011
367765499464,6cyclictest23-21ksoftirqd/110:07:441
3677654994631,11cyclictest275-21dbus-daemon07:55:001
3677654994614,26cyclictest48-21kcompactd008:20:501
3677649994624,15cyclictest275-21dbus-daemon07:25:000
367765499456,7cyclictest23-21ksoftirqd/111:20:041
367765499448,8cyclictest23-21ksoftirqd/107:50:021
367765499447,8cyclictest0-21swapper/109:01:261
3677654994433,6cyclictest309-21in:imuxsock07:20:001
3677654994412,5cyclictest0-21swapper/107:13:481
3677649994426,14cyclictest3680670-21sshd08:06:100
3677660994322,17cyclictest3677687-21latency_hist07:10:013
367765499438,9cyclictest0-21swapper/107:26:111
367765499438,8cyclictest0-21swapper/108:51:531
367765499428,8cyclictest0-21swapper/107:36:071
3677654994227,12cyclictest1-21systemd07:45:011
3677654994227,12cyclictest1-21systemd07:45:001
367765499421,8cyclictest0-21swapper/108:42:441
367765499418,8cyclictest0-21swapper/107:22:411
3677654994112,5cyclictest0-21swapper/108:57:131
367765499408,5cyclictest0-21swapper/108:04:141
367765499407,7cyclictest0-21swapper/109:07:551
367765499407,7cyclictest0-21swapper/109:07:541
3677654994032,6cyclictest23-21ksoftirqd/108:50:021
367765499391,6cyclictest0-21swapper/108:05:441
3677649993931,4cyclictest16950irq/224-800000008:05:000
367765499388,7cyclictest0-21swapper/108:30:431
3677649993729,5cyclictest19350irq/34-mmc108:28:570
3677649993722,11cyclictest3679651-21sshd07:49:200
3677656993426,6cyclictest30-21ksoftirqd/207:10:012
3677654993119,9cyclictest3677693-21latency_hist07:10:021
3677649992925,2cyclictest0-21swapper/007:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional