You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 18:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 09, 2026 12:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
356192699465438,24cyclictest3665990-21turbostat12:00:003
356192699463439,8cyclictest3576554-21turbostat07:54:553
356192699462455,4cyclictest3669504-21turbostat12:14:563
356192699462455,4cyclictest3620795-21turbostat09:59:563
356192699462440,19cyclictest3667768-21turbostat12:04:593
356192699459452,4cyclictest3574832-21turbostat07:49:563
356192499456448,5cyclictest3624559-21turbostat10:09:542
356192699454447,4cyclictest3646896-21turbostat11:05:003
356192699453445,5cyclictest3619067-21turbostat09:50:003
356192499453435,15cyclictest3672964-21turbostat12:24:562
356192699452445,4cyclictest3606649-21turbostat09:15:003
356192699452442,7cyclictest3628329-21turbostat10:19:553
356192699451443,5cyclictest3653874-21turbostat11:25:003
356192699450442,5cyclictest3674696-21turbostat12:25:003
356192699449442,4cyclictest3565820-21turbostat07:20:013
356192699446438,5cyclictest3672964-21turbostat12:20:013
356192699444437,4cyclictest3596254-21turbostat08:49:553
356192699443436,4cyclictest3567538-21turbostat07:29:543
356192699443435,5cyclictest3631788-21turbostat10:29:563
356192699440432,5cyclictest3644853-21turbostat11:00:003
356192699440416,21cyclictest3599716-21turbostat08:54:593
356192699435428,4cyclictest3581813-21turbostat08:09:563
356192699435424,7cyclictest3626597-21turbostat10:14:553
356192699433426,4cyclictest3611833-21turbostat09:34:553
356192699433408,22cyclictest3563599-21turbostat07:15:003
356192699430423,4cyclictest3565820-21turbostat07:24:563
356192699425418,4cyclictest3678161-21turbostat12:39:563
356192499425402,20cyclictest3597996-21turbostat08:54:562
356192699424399,22cyclictest3669504-21turbostat12:10:013
356192699421412,6cyclictest3594536-21turbostat08:40:003
356192699420413,4cyclictest3630058-21turbostat10:24:563
356192499420402,15cyclictest3646896-21turbostat11:09:552
356192499420402,15cyclictest3646896-21turbostat11:09:552
356192699419394,22cyclictest3615311-21turbostat09:44:553
356192499419412,4cyclictest3622836-21turbostat10:04:562
356192699417410,4cyclictest3601459-21turbostat09:00:003
356192699414407,4cyclictest3604920-21turbostat09:10:013
356192699413405,5cyclictest3574832-21turbostat07:45:013
356192699410402,5cyclictest3636977-21turbostat10:40:013
356192699407399,5cyclictest3674696-21turbostat12:29:543
356192699405396,6cyclictest3620795-21turbostat09:55:013
356191699403396,4cyclictest3642778-21turbostat10:59:550
356192699400393,4cyclictest3650398-21turbostat11:19:543
356192699397390,4cyclictest3613578-21turbostat09:39:553
356192699395387,5cyclictest3636977-21turbostat10:44:563
356192699393385,5cyclictest3657333-21turbostat11:35:003
356192499393386,4cyclictest3660795-21turbostat11:49:552
356192499393371,8cyclictest3573103-21turbostat07:44:562
356192699391384,4cyclictest3585577-21turbostat08:19:553
356192699391383,5cyclictest3581813-21turbostat08:05:013
356192699391383,5cyclictest3581813-21turbostat08:05:003
356192699388381,4cyclictest3589338-21turbostat08:29:563
356192499388381,4cyclictest3635248-21turbostat10:39:542
356192499387379,5cyclictest3592808-21turbostat08:39:552
356192699385378,4cyclictest3594536-21turbostat08:44:553
356191699380372,5cyclictest3587308-21turbostat08:24:540
356192699379370,6cyclictest3655612-21turbostat11:30:013
356192699378371,4cyclictest3664264-21turbostat11:55:013
356192699373367,3cyclictest3580068-21turbostat08:00:003
356192699373366,4cyclictest3583848-21turbostat08:14:553
356192699373365,5cyclictest3648629-21turbostat11:14:553
356192699371364,4cyclictest3638714-21turbostat10:49:563
356192699370363,4cyclictest3662531-21turbostat11:50:013
356192699364356,5cyclictest3611833-21turbostat09:30:003
356192699364356,5cyclictest3601459-21turbostat09:04:553
356192499364358,3cyclictest3563599-21turbostat07:19:552
356192499362355,4cyclictest3676429-21turbostat12:34:552
356192699361353,5cyclictest3635248-21turbostat10:34:593
356192699360353,4cyclictest3610109-21turbostat09:25:003
356192499360354,3cyclictest3653874-21turbostat11:29:552
356192699358349,6cyclictest3592808-21turbostat08:35:003
356192499355348,4cyclictest3608377-21turbostat09:24:542
356191999348342,3cyclictest3619067-21turbostat09:54:541
356192699338331,4cyclictest3657333-21turbostat11:39:553
356192699338330,5cyclictest3571311-21turbostat07:39:553
356192699337329,5cyclictest3659064-21turbostat11:44:543
356192699337329,5cyclictest3659064-21turbostat11:44:543
356192699336329,4cyclictest3608377-21turbostat09:20:003
356192499327321,3cyclictest3633519-21turbostat10:34:552
356192499326315,8cyclictest3640746-21turbostat10:54:552
356192499322316,3cyclictest3591071-21turbostat08:34:552
356192699318312,3cyclictest3648629-21turbostat11:10:003
356192699318312,3cyclictest3648629-21turbostat11:10:003
356192699316310,3cyclictest3569280-21turbostat07:34:553
356192699316308,5cyclictest3642778-21turbostat10:54:593
356192699223214,6cyclictest3624559-21turbostat10:04:593
356192699176167,6cyclictest3626597-21turbostat10:09:593
3560315211076,23sleep00-21swapper/007:05:070
3561603210976,23sleep10-21swapper/107:07:331
3561723210874,23sleep20-21swapper/207:09:062
3561563210673,23sleep30-21swapper/307:07:033
3561926999891,4cyclictest3589338-21turbostat08:25:003
3561926999689,4cyclictest3678161-21turbostat12:35:003
3561924999689,4cyclictest3606649-21turbostat09:19:552
36420482860,7sleep04497-21PK-Backend10:50:260
36271182770,4sleep03627119-21cat10:10:150
197399660,5rtkit-daemon0-21swapper/112:22:581
36141072650,6sleep10-21swapper/109:35:141
35768222600,17sleep126-21ksoftirqd/107:50:101
3561924995815,16cyclictest3666232-21systemd-udevd12:00:062
3561924995515,20cyclictest3615003-21taskset09:36:192
356192499547,27cyclictest4497-21PK-Backend11:54:052
3561924995414,15cyclictest3670993-21taskset12:12:092
3561916995432,7cyclictest3592799-21apt08:35:000
356192499539,26cyclictest0-21swapper/210:16:062
3561924995314,20cyclictest1137-21runrttasks12:09:402
3561924995314,13cyclictest3672699-21taskset12:16:532
36040292520,7sleep20-21swapper/209:05:212
35770892520,15sleep0131ksoftirqd/007:50:120
3561924995114,14cyclictest3583742-21taskset08:08:582
3561924995111,16cyclictest1137-21runrttasks07:56:302
36789152500,12sleep23561910-21cyclictest12:35:202
3561924995013,15cyclictest3631486-21taskset10:21:192
3561924995013,11cyclictest3611125-21ntp_states09:25:252
3561924994911,17cyclictest1137-21runrttasks11:14:552
356192499489,18cyclictest1137-21runrttasks09:13:372
356192499489,14cyclictest0-21swapper/208:00:572
356192499489,14cyclictest0-21swapper/208:00:562
3561924994821,22cyclictest3588227-21/usr/sbin/munin08:20:172
3561924994813,31cyclictest3643805-21kernelversion10:55:212
36238752470,5sleep1251rcuc/110:00:261
356192499479,15cyclictest3676352-21taskset12:29:212
356192499477,15cyclictest3620750-21runrttasks09:54:382
3561924994715,27cyclictest3645750-21df_inode11:00:162
3561924994715,27cyclictest3618684-21users09:45:292
3561924994715,10cyclictest3638382-21taskset10:40:582
3561924994714,28cyclictest3641117-21latency_hist10:50:022
3561924994713,30cyclictest3617454-21ls09:45:022
356192499469,14cyclictest0-21swapper/211:58:142
356192499469,14cyclictest0-21swapper/207:52:202
3561924994618,23cyclictest3567531-21cron07:25:002
3561924994611,15cyclictest1137-21runrttasks07:37:252
3561924994611,13cyclictest1137-21runrttasks08:29:052
197399460,31rtkit-daemon3582334-21apt08:05:111
356192499458,16cyclictest0-21swapper/211:17:572
356192499458,15cyclictest3660616-21runrttasks11:42:562
356192499458,15cyclictest3660616-21runrttasks11:42:562
356192499458,14cyclictest0-21swapper/209:31:562
36041582440,7chrt4497-21PK-Backend09:05:230
35862942440,7chrt4497-21PK-Backend08:15:200
356192499449,12cyclictest3601401-21taskset08:59:322
356192499449,10cyclictest0-21swapper/210:25:412
356192499448,14cyclictest4497-21PK-Backend08:12:292
356192499448,13cyclictest0-21swapper/209:59:252
3561924994410,10cyclictest3595997-21taskset08:41:462
356192499439,11cyclictest3563422-21taskset07:12:592
356192499438,11cyclictest0-21swapper/207:30:502
3561924994315,24cyclictest923-21dbus-daemon10:15:002
3561924994314,11cyclictest438-20systemd-journal11:39:552
36359842420,7sleep10-21swapper/110:35:191
3561924994217,21cyclictest3652546-21tr11:20:132
356191999423,17cyclictest3646592-21taskset11:01:591
3561919994225,12cyclictest923-21dbus-daemon11:12:391
197399420,23rtkit-daemon0-21swapper/111:42:011
197399420,23rtkit-daemon0-21swapper/111:42:011
356191999417,30cyclictest3638052-21grep10:40:251
356191999414,17cyclictest3571216-21taskset07:34:001
356191999413,18cyclictest3628152-21taskset10:13:011
356191999413,18cyclictest3566055-21taskset07:20:081
356191999412,17cyclictest3608233-21taskset09:18:201
3561919994120,17cyclictest3586295-21/usr/sbin/munin08:15:211
3561919994114,22cyclictest3641912-40lsb_release10:50:241
3561919994111,7cyclictest36561332expr11:30:141
356191999411,14cyclictest3571541-21taskset07:35:031
35940762400,6sleep1251rcuc/108:35:291
35820062400,10sleep0161rcuc/008:05:020
35820062400,10sleep0161rcuc/008:05:020
3561924994018,17cyclictest3575497-21hddtemp_smartct07:45:182
3561924994014,22cyclictest3586461-21cat08:15:232
356191999407,10cyclictest0-21swapper/110:58:571
356191999405,8cyclictest26-21ksoftirqd/110:05:081
356191999403,14cyclictest3629874-21taskset10:17:521
3561916994028,8cyclictest3563597-21ls07:15:000
356191999397,12cyclictest3612112-21taskset09:30:111
3561919993922,13cyclictest3585410-21sshd08:13:091
3561919993922,12cyclictest438-20systemd-journal10:25:001
356191999392,17cyclictest3622478-21taskset09:56:111
197399390,25rtkit-daemon0-21swapper/111:22:241
36719792380,13sleep126-21ksoftirqd/112:15:191
3561924993815,18cyclictest3655843-21/usr/sbin/munin11:30:302
356191999389,11cyclictest3668008-21taskset12:05:101
3561919993822,11cyclictest438-20systemd-journal07:30:001
356191999382,16cyclictest3595909-21taskset08:40:361
356191999382,16cyclictest3574655-21taskset07:42:571
3561919993817,17cyclictest976-21rs:main1
3561916993817,9cyclictest3625531-21if_enp2s010:05:200
197399380,25rtkit-daemon0-21swapper/109:24:511
36757252370,8sleep04497-21PK-Backend12:25:240
3561924993718,15cyclictest3568687-21python307:25:272
356191999379,6cyclictest36471532acpi11:05:101
356191999379,6cyclictest36471532acpi11:05:101
356191999375,12cyclictest3633158-21taskset10:25:371
356191999371,15cyclictest3671216-21sh12:15:001
3561916993725,8cyclictest13-21ksoftirqd/007:56:370
3561916993719,13cyclictest3647524-21df11:05:140
3561916993719,13cyclictest3647524-21df11:05:140
356191699371,14cyclictest4497-21PK-Backend11:18:040
197399370,25rtkit-daemon0-21swapper/108:51:451
197399370,25rtkit-daemon0-21swapper/108:33:161
197399370,24rtkit-daemon0-21swapper/109:05:221
197399370,24rtkit-daemon0-21swapper/108:47:241
3561924993617,14cyclictest3602465-21/usr/sbin/munin09:00:232
356192499361,13cyclictest4497-21PK-Backend08:47:522
356191999366,7cyclictest0-21swapper/112:03:451
356191999364,14cyclictest26-21ksoftirqd/109:00:161
356191999363,13cyclictest3676387-21taskset12:29:491
3561919993621,11cyclictest3588806-21threads08:20:281
3561919993620,12cyclictest3663513-21awk11:50:221
356191699362,7cyclictest3665990-21turbostat11:59:590
3561916993621,11cyclictest937-21polkitd12:00:060
3561916993621,11cyclictest3615298-21grep09:40:000
3561916993620,12cyclictest3596500-21sshd08:45:080
356191699361,14cyclictest4497-21PK-Backend10:44:410
356191699361,14cyclictest4497-21PK-Backend08:44:070
356191999359,8cyclictest0-21swapper/107:55:591
356191999356,8cyclictest0-21swapper/109:10:231
356191999354,8cyclictest26-21ksoftirqd/112:33:231
356191999353,11cyclictest3611551-21taskset09:26:281
3561919993526,6cyclictest3618254-21tail09:45:221
3561919993523,8cyclictest2069-21gnome-shell11:37:081
356191999352,10cyclictest26-21ksoftirqd/112:36:291
356191999352,10cyclictest0-21swapper/111:16:401
3561919993519,12cyclictest3576092-21timerandwakeup07:45:281
3561919993519,11cyclictest3590374-21open_inodes08:25:251
3561916993517,13cyclictest3584696-21memory08:10:220
3561916993516,15cyclictest3658682-21users11:35:300
3561916993514,18cyclictest3640526-40lsb_release10:47:300
356191699351,15cyclictest4497-21PK-Backend11:48:480
356191999348,22cyclictest3661203-21cpuspeed_turbos11:45:141
356191999344,9cyclictest26-21ksoftirqd/110:30:561
356191999344,8cyclictest26-21ksoftirqd/111:28:231
356191999343,9cyclictest26-21ksoftirqd/108:00:081
356191999343,9cyclictest26-21ksoftirqd/108:00:081
3561919993423,7cyclictest3600159-21tr08:55:141
3561916993420,10cyclictest923-21dbus-daemon08:05:320
356191699341,9cyclictest4497-21PK-Backend11:20:590
356191699341,9cyclictest4497-21PK-Backend10:28:580
3561916993418,12cyclictest3603231-21sed09:05:020
3561916993417,13cyclictest3679802-21sshd12:39:130
356191699341,4cyclictest4497-21PK-Backend11:29:010
356191699341,29cyclictest4497-21PK-Backend11:40:220
356191699341,29cyclictest4497-21PK-Backend11:40:210
356191999334,8cyclictest26-21ksoftirqd/109:43:191
356191999333,8cyclictest26-21ksoftirqd/110:45:231
356191999331,29cyclictest0-21swapper/107:16:251
356191999331,10cyclictest0-21swapper/107:12:541
3561916993323,6cyclictest3571320-21ls07:35:000
3561916993322,7cyclictest3574823-21cron07:45:010
356191699331,9cyclictest4497-21PK-Backend11:03:570
356191699331,9cyclictest4497-21PK-Backend09:34:010
3561916993319,10cyclictest3664249-21cron11:55:010
3561916993317,12cyclictest3565218-21sensors_temp07:15:260
3561916993317,12cyclictest1033-21gdbus11:12:400
356191699321,9cyclictest4497-21PK-Backend11:32:260
356191699321,9cyclictest4497-21PK-Backend10:36:120
356191699321,8cyclictest4497-21PK-Backend07:28:010
3561916993217,11cyclictest3609498-21awk09:20:260
3561916993217,11cyclictest3576264-21seq07:46:190
3561916993215,12cyclictest438-20systemd-journal08:25:020
356191699321,13cyclictest4497-21PK-Backend12:34:520
356191699321,13cyclictest4497-21PK-Backend10:23:080
356191699321,10cyclictest4497-21PK-Backend09:11:190
3561916993123,5cyclictest4497-21PK-Backend07:22:080
3561916993118,9cyclictest3674696-21turbostat12:24:590
3561916993118,9cyclictest3630102-21sed10:20:020
3561916993115,12cyclictest3669091-21taskset12:05:290
356191699311,4cyclictest4497-21PK-Backend08:51:220
356191699311,13cyclictest4497-21PK-Backend09:50:440
356191699311,10cyclictest4497-21PK-Backend10:31:160
356191699304,11cyclictest13-21ksoftirqd/009:44:070
3561916993019,6cyclictest13-21ksoftirqd/009:48:250
3561916993013,12cyclictest3572032-21cat07:35:190
3561916992922,4cyclictest4497-21PK-Backend08:39:180
356191699291,25cyclictest4497-21PK-Backend09:56:260
356191699291,24cyclictest4497-21PK-Backend12:10:280
3561916992910,5cyclictest141rcu_preempt09:15:090
3561916992821,4cyclictest4497-21PK-Backend12:16:160
3561916992821,4cyclictest4497-21PK-Backend09:27:050
356191699281,24cyclictest4497-21PK-Backend08:55:510
356191699271,10cyclictest4497-21PK-Backend10:01:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional