You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 08:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 19, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36182882810,7sleep14497-21PK-Backend20:10:231
359397427740,9sleep30-21swapper/319:05:073
197399700,58rtkit-daemon0-21swapper/219:05:522
197399690,9rtkit-daemon1972-21rtkit-daemon22:08:213
197399690,4rtkit-daemon0-21swapper/019:47:460
359496026547,10sleep10-21swapper/119:05:291
359419226442,14sleep00-21swapper/019:05:160
197399630,8rtkit-daemon1972-21rtkit-daemon22:52:173
197399620,9rtkit-daemon1972-21rtkit-daemon00:25:233
197399620,8rtkit-daemon1972-21rtkit-daemon19:58:123
197399600,9rtkit-daemon1972-21rtkit-daemon19:22:523
197399590,8rtkit-daemon1972-21rtkit-daemon21:26:233
36938942580,5sleep20-21swapper/223:45:142
197399580,8rtkit-daemon1972-21rtkit-daemon22:59:442
36540112570,21sleep234-21ksoftirqd/221:50:202
197399570,9rtkit-daemon1972-21rtkit-daemon21:11:152
197399550,9rtkit-daemon1972-21rtkit-daemon23:15:372
197399550,9rtkit-daemon1972-21rtkit-daemon21:18:563
197399550,8rtkit-daemon1972-21rtkit-daemon21:21:183
197399520,8rtkit-daemon1972-21rtkit-daemon22:15:332
197399520,8rtkit-daemon1972-21rtkit-daemon19:52:352
197399520,8rtkit-daemon1972-21rtkit-daemon00:11:172
36903802510,5chrt4497-21PK-Backend23:35:213
36903802510,5chrt4497-21PK-Backend23:35:213
3595624995127,9cyclictest34-21ksoftirqd/222:25:092
3595624995127,9cyclictest34-21ksoftirqd/222:25:092
3595624995119,13cyclictest1-21systemd23:40:302
197399510,8rtkit-daemon1972-21rtkit-daemon21:29:122
197399510,8rtkit-daemon1972-21rtkit-daemon21:01:002
37106572460,8sleep013-21ksoftirqd/000:30:310
36199592450,7sleep14497-21PK-Backend20:15:221
197399450,37rtkit-daemon1074-21lldpd23:02:023
359562499443,9cyclictest34-21ksoftirqd/220:40:002
197399440,7rtkit-daemon3697999-21sed23:55:202
36660532430,5chrt0-21swapper/022:25:150
36660532430,5chrt0-21swapper/022:25:140
359562899432,26cyclictest0-21swapper/321:15:023
37042032420,5chrt0-21swapper/000:15:010
35990522420,5chrt0-21swapper/019:15:280
35964312420,3sleep10-21swapper/119:10:221
36015682410,6sleep20-21swapper/219:25:122
3595628994131,6cyclictest3611928-21sh19:55:003
359562899412,7cyclictest141rcu_preempt23:40:303
3595624994125,6cyclictest34-21ksoftirqd/223:10:152
3595624994125,6cyclictest34-21ksoftirqd/223:10:142
3595622994125,12cyclictest923-21dbus-daemon21:23:061
3595619994119,18cyclictest3656853-21date22:00:010
3595628994025,11cyclictest923-21dbus-daemon22:29:573
3595628994025,11cyclictest923-21dbus-daemon22:29:563
3595622994023,12cyclictest3664623-21ntp_kernel_err22:20:231
3595622994022,14cyclictest26-21ksoftirqd/123:03:071
3595622994015,20cyclictest3651871-21cat21:45:141
3595622994012,7cyclictest36958012tr23:50:141
3595619994024,11cyclictest438-20systemd-journal19:52:100
3595619994020,6cyclictest13-21ksoftirqd/022:12:110
197399400,33rtkit-daemon4497-21PK-Backend22:32:052
36643112390,15chrt34-21ksoftirqd/222:20:182
3595628993922,13cyclictest42-21ksoftirqd/323:49:593
3595628993921,14cyclictest42-21ksoftirqd/319:50:013
3595624993928,8cyclictest34-21ksoftirqd/219:14:562
3595622993921,14cyclictest26-21ksoftirqd/123:48:191
3595619993928,7cyclictest3638805-21grep21:10:000
3595619993924,11cyclictest3681006-21dump-pmu-power23:10:010
3595619993922,6cyclictest13-21ksoftirqd/023:46:320
3595619993921,14cyclictest3605734-21ntp_states19:35:260
359562899382,9cyclictest141rcu_preempt19:35:023
3595628993826,7cyclictest3657666-21grep22:00:223
3595628993814,7cyclictest3667930-21kernelversion22:30:213
3595628993813,12cyclictest141rcu_preempt20:45:153
3595624993822,12cyclictest3707592-21sleep00:25:002
3595624993822,12cyclictest3629843-21sshd20:41:092
3595624993821,13cyclictest34-21ksoftirqd/219:55:152
3595624993821,12cyclictest438-20systemd-journal21:33:102
3595624993814,14cyclictest34-21ksoftirqd/220:30:242
359562299382,32cyclictest3617254-40lsb_release20:08:061
3595622993821,13cyclictest3603994-21ntp_states19:30:231
3595622993821,13cyclictest26-21ksoftirqd/119:55:341
3595622993820,6cyclictest26-21ksoftirqd/120:39:081
3595622993820,14cyclictest26-21ksoftirqd/121:09:431
3595622993817,17cyclictest3600658-21python319:20:251
3595619993825,9cyclictest3596214-21sort19:10:190
3595619993823,11cyclictest3712752-21cron00:39:590
3595619993822,11cyclictest438-20systemd-journal23:19:550
3595619993821,6cyclictest13-21ksoftirqd/021:02:480
359561999381,34cyclictest0-21swapper/021:18:320
197399380,29rtkit-daemon42-21ksoftirqd/300:03:583
37051242370,6chrt0-21swapper/300:15:243
36971962370,5chrt0-21swapper/323:55:003
36701672370,3sleep10-21swapper/122:35:311
359562899375,12cyclictest42-21ksoftirqd/320:40:353
3595628993720,6cyclictest42-21ksoftirqd/320:40:003
3595628993719,6cyclictest42-21ksoftirqd/319:45:003
3595628993719,13cyclictest438-20systemd-journal20:15:003
3595628993712,21cyclictest3620886-21latency_hist20:20:023
3595624993727,7cyclictest1137-21runrttasks22:35:202
3595624993726,7cyclictest3651220-21ls21:44:592
3595624993719,14cyclictest3614681-21ntp_states20:00:252
3595624993719,13cyclictest438-20systemd-journal00:10:002
3595624993717,6cyclictest141rcu_preempt21:05:152
3595624993714,14cyclictest34-21ksoftirqd/223:35:262
3595624993714,14cyclictest34-21ksoftirqd/223:35:252
3595622993721,12cyclictest3661660-21seq22:11:591
3595622993721,12cyclictest3606430-21cron19:40:001
3595622993720,13cyclictest3665908-21expr22:25:141
3595622993720,13cyclictest3665908-21expr22:25:131
3595622993720,13cyclictest3625517-21memory20:30:241
3595622993718,14cyclictest3698309-21python323:55:271
3595622993716,17cyclictest2345-21gdbus21:39:571
3595619993723,10cyclictest3670616-21turbostat22:40:000
3595619993720,12cyclictest438-20systemd-journal00:17:000
359561999371,5cyclictest141rcu_preempt23:15:010
359561999371,5cyclictest141rcu_preempt23:15:010
359561999371,32cyclictest4497-21PK-Backend19:57:110
359561999371,31cyclictest3692032-21ld-2.23.so23:40:160
359562899366,9cyclictest42-21ksoftirqd/300:40:013
359562899366,15cyclictest0-21swapper/323:05:323
359562899365,9cyclictest42-21ksoftirqd/320:05:003
3595628993626,7cyclictest3686192-21cron23:24:593
3595628993620,11cyclictest438-20systemd-journal22:39:103
3595628993618,13cyclictest3605718-21ntpq19:35:243
3595624993627,6cyclictest3672335-21dump-pmu-power22:45:002
3595624993626,6cyclictest3633580-21NetworkManager20:54:442
3595624993626,6cyclictest3633580-21NetworkManager20:54:432
3595624993621,11cyclictest3635336-21cron21:00:002
3595624993620,5cyclictest34-21ksoftirqd/223:35:012
3595624993619,13cyclictest3708211-21df_inode00:25:142
359562299362,30cyclictest3598425-21df_inode19:15:151
3595622993621,11cyclictest3704119-21basename00:15:001
3595622993620,11cyclictest438-20systemd-journal00:25:101
359562299361,9cyclictest3647835-21latency_hist21:35:021
3595622993619,13cyclictest3701673-21ntp_states00:05:301
3595622993619,13cyclictest3607337-21df19:40:151
3595622993618,14cyclictest3641392-21memory21:15:221
3595622993617,6cyclictest26-21ksoftirqd/120:42:421
3595622993612,6cyclictest36025322timerwakeupswit19:25:281
359561999361,30cyclictest3647771-21turbostat21:34:590
359561999361,13cyclictest4497-21PK-Backend23:20:340
197399360,29rtkit-daemon0-21swapper/321:56:463
359562899357,13cyclictest0-21swapper/320:05:133
359562899356,9cyclictest42-21ksoftirqd/320:28:563
359562899356,9cyclictest42-21ksoftirqd/320:28:563
359562899355,10cyclictest42-21ksoftirqd/319:10:393
359562899353,10cyclictest42-21ksoftirqd/320:22:523
359562899352,12cyclictest141rcu_preempt00:00:023
3595628993519,12cyclictest3639020-21sed21:10:023
3595628993518,13cyclictest42-21ksoftirqd/320:55:123
359562899351,30cyclictest4497-21PK-Backend20:52:333
359562899351,30cyclictest4497-21PK-Backend20:52:323
3595624993524,7cyclictest3704506-21tr00:15:142
3595624993521,10cyclictest34-21ksoftirqd/223:20:182
3595624993520,11cyclictest34-21ksoftirqd/219:15:272
3595624993519,12cyclictest1054-21snmpd19:38:142
3595624993518,13cyclictest1-21systemd20:50:012
3595622993525,6cyclictest3634031-21tr20:55:141
3595622993524,7cyclictest3671037-21tr22:40:141
3595622993523,8cyclictest3706630-21latency00:20:221
3595622993520,11cyclictest3631966-21wc20:50:011
359562299351,9cyclictest4497-21PK-Backend23:40:151
3595622993519,12cyclictest3681515-21expr23:10:161
3595622993519,12cyclictest3681515-21expr23:10:151
3595622993519,12cyclictest3653924-21fw_packets21:50:171
3595622993519,11cyclictest3674635-21expr22:50:141
3595622993518,13cyclictest3623780-21memory20:25:221
3595622993518,13cyclictest3623780-21memory20:25:221
3595622993518,13cyclictest3622704-40apt-check20:22:521
3595619993521,10cyclictest3633603-21cron20:55:000
3595619993521,10cyclictest3633603-21cron20:55:000
3595619993520,11cyclictest923-21dbus-daemon21:40:010
3595619993520,11cyclictest3631408-21sed20:45:290
3595619993519,12cyclictest976-21rs:main0
3595619993519,12cyclictest3697002-21sshd23:53:070
3595619993516,14cyclictest3701285-21df00:05:190
359561999351,31cyclictest4497-21PK-Backend21:47:500
359561999351,31cyclictest4497-21PK-Backend21:12:480
197399350,28rtkit-daemon0-21swapper/319:15:073
3595628993423,7cyclictest3646480-21tr21:30:133
3595628993423,6cyclictest42-21ksoftirqd/323:34:573
3595628993422,8cyclictest3676571-21irqstats22:55:213
3595628993420,10cyclictest3665576-21ls22:25:023
3595628993419,11cyclictest42-21ksoftirqd/323:25:183
359562499342,8cyclictest924-21NetworkManager23:52:362
3595624993423,7cyclictest3655928-21grep21:55:222
3595624993419,11cyclictest3623903-21awk20:25:242
3595624993419,11cyclictest3623903-21awk20:25:242
3595624993419,11cyclictest3603966-21awk19:30:222
3595624993418,5cyclictest34-21ksoftirqd/221:35:152
3595624993418,13cyclictest34-21ksoftirqd/200:30:272
3595624993418,11cyclictest3620885-21sh20:20:022
3595624993417,12cyclictest438-20systemd-journal23:05:012
359562499341,29cyclictest3607860-21awk19:40:282
359562299342,12cyclictest20691gnome-shell21:03:591
3595622993420,10cyclictest3685387-21sed23:20:241
3595622993419,11cyclictest3668156-21ntpq22:30:261
3595622993419,11cyclictest3655836-21load21:55:211
3595622993419,11cyclictest3645954-21sshd21:29:111
3595622993419,11cyclictest3633223-21cat20:50:321
3595622993419,11cyclictest3633223-21cat20:50:321
3595622993418,12cyclictest3688608-21which23:30:191
3595622993418,12cyclictest3673681-21cat22:45:311
3595622993418,12cyclictest3663180-21cat22:15:301
3595622993418,11cyclictest3709534-21seq00:30:021
3595622993417,5cyclictest26-21ksoftirqd/100:02:101
3595622993417,13cyclictest3683006-21apt23:15:111
359562299341,11cyclictest3711583-21expr00:35:141
3595619993424,6cyclictest3604711-21latency_hist19:34:590
3595619993419,11cyclictest923-21dbus-daemon22:55:000
3595619993419,11cyclictest923-21dbus-daemon22:20:010
3595619993419,11cyclictest3708005-21cpuspeed_turbos00:25:130
3595619993419,11cyclictest3699774-21load00:00:230
3595619993419,11cyclictest3658264-21seq22:02:500
3595619993417,13cyclictest976-21rs:main0
3595619993417,13cyclictest3645844-21seq21:27:480
3595619993417,12cyclictest3623167-21sshd20:25:100
3595619993417,12cyclictest3623167-21sshd20:25:090
3595619993416,14cyclictest1975-21dbus-daemon20:08:080
359561999341,4cyclictest4497-21PK-Backend20:38:180
359561999341,30cyclictest4497-21PK-Backend22:09:190
359561999341,30cyclictest4497-21PK-Backend20:03:580
359561999341,30cyclictest0-21swapper/022:34:130
359561999341,10cyclictest4497-21PK-Backend20:43:410
197399340,28rtkit-daemon0-21swapper/223:01:352
197399340,28rtkit-daemon0-21swapper/220:13:192
197399340,27rtkit-daemon0-21swapper/222:54:222
197399340,25rtkit-daemon1972-21rtkit-daemon21:36:113
197399340,25rtkit-daemon1972-21rtkit-daemon21:22:442
197399340,25rtkit-daemon1941-21systemd21:48:063
359562899334,10cyclictest42-21ksoftirqd/323:11:593
359562899334,10cyclictest42-21ksoftirqd/323:11:593
3595628993319,10cyclictest3672865-21ls22:45:153
359562899331,4cyclictest4497-21PK-Backend00:12:413
359562899331,12cyclictest3702140-21sshd00:07:103
359562899331,11cyclictest141rcu_preempt19:25:143
3595624993327,4cyclictest0-21swapper/222:10:302
3595624993323,6cyclictest34-21ksoftirqd/221:45:072
3595624993322,6cyclictest34-21ksoftirqd/223:25:162
3595624993320,10cyclictest34-21ksoftirqd/219:45:182
3595624993318,12cyclictest34-21ksoftirqd/222:00:592
3595624993317,5cyclictest34-21ksoftirqd/222:05:292
3595624993317,12cyclictest3601236-21cron19:25:002
3595622993318,10cyclictest3613923-21find20:00:111
3595622993317,12cyclictest3691152-21seq23:37:071
3595622993317,12cyclictest3691152-21seq23:37:061
3595622993317,12cyclictest3676449-21expr22:55:161
3595622993317,12cyclictest3659063-21expr22:05:151
3595622993317,12cyclictest3657676-21netstat22:00:241
3595622993317,12cyclictest3650067-21cat21:40:161
3595622993317,11cyclictest3611204-21ntp_kernel_pll_19:50:241
3595619993323,6cyclictest3626378-21cron20:35:000
3595619993319,10cyclictest3617849-21grep20:10:140
3595619993318,10cyclictest3642270-21apt-get21:20:030
359561999331,4cyclictest4497-21PK-Backend22:47:380
3595619993313,15cyclictest3664277-21df22:20:160
359561999331,29cyclictest4497-21PK-Backend23:02:510
359561999331,29cyclictest4497-21PK-Backend21:41:460
359561999331,10cyclictest0-21swapper/023:57:330
359562899322,11cyclictest3670875-21acpi22:40:113
3595628993219,9cyclictest3651214-21cron21:45:003
359562899321,28cyclictest4497-21PK-Backend21:52:363
359562899321,28cyclictest4497-21PK-Backend20:32:133
359562899321,28cyclictest4497-21PK-Backend00:24:163
359562899321,10cyclictest1054-21snmpd22:10:463
3595624993223,6cyclictest3642403-21apt-key21:20:002
3595624993221,6cyclictest34-21ksoftirqd/220:05:042
3595624993220,8cyclictest34-21ksoftirqd/200:35:242
359562499321,4cyclictest0-21swapper/220:17:372
359562499321,28cyclictest0-21swapper/222:48:032
359562299325,22cyclictest3688131-21cat23:30:021
3595622993217,11cyclictest3704806-21cat00:15:201
3595622993214,5cyclictest26-21ksoftirqd/123:05:571
359562299321,28cyclictest4497-21PK-Backend21:10:191
359561999324,5cyclictest141rcu_preempt22:40:140
3595619993222,6cyclictest3688395-21cpuspeed_turbos23:30:140
3595619993220,8cyclictest2069-21gnome-shell21:51:090
3595619993217,11cyclictest976-21rs:main0
3595619993217,11cyclictest3690050-21cpuspeed_turbos23:35:140
3595619993217,11cyclictest3690050-21cpuspeed_turbos23:35:140
3595619993216,12cyclictest3607532-21sed19:40:200
3595619993216,11cyclictest438-20systemd-journal20:59:110
3595628993124,4cyclictest4497-21PK-Backend00:30:123
359562499312,25cyclictest3699936-21ntp_states00:00:272
359561999318,6cyclictest141rcu_preempt20:20:220
3595619993117,10cyclictest3600516-21tr19:20:210
3595619993116,11cyclictest3602170-21grep19:25:210
3595622993017,9cyclictest3610082-21seq19:48:361
3595619993020,5cyclictest13-21ksoftirqd/023:25:200
197399300,23rtkit-daemon4497-21PK-Backend23:15:443
359562899293,10cyclictest141rcu_preempt22:15:233
3595628992717,6cyclictest36369392sleep321:03:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional