You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 17:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 16, 2026 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
337633099507499,5cyclictest3489768-21turbostat12:25:003
337633099465457,5cyclictest3441671-21turbostat10:10:003
337633099465457,5cyclictest3416499-21turbostat09:00:003
337633099465455,6cyclictest3450635-21turbostat10:35:003
337633099464457,4cyclictest3425200-21turbostat09:29:553
337633099463456,4cyclictest3472050-21turbostat11:39:563
337633099463455,5cyclictest3380202-21turbostat07:19:593
337633099462453,6cyclictest3452366-21turbostat10:44:553
337633099461454,4cyclictest3390190-21turbostat07:49:543
337632599461454,4cyclictest3493243-21turbostat12:39:552
337633099459451,4cyclictest3470280-21turbostat11:30:003
337633099458451,4cyclictest3437915-21turbostat10:04:563
337633099458451,4cyclictest3437915-21turbostat10:04:553
337633099457450,4cyclictest3466507-21turbostat11:24:553
337633099456449,4cyclictest3397161-21turbostat08:09:553
337632599455446,6cyclictest3439947-21turbostat10:09:552
337632599455433,19cyclictest3420002-21turbostat09:14:552
337633099452443,6cyclictest3475810-21turbostat11:45:003
337632599449427,19cyclictest3423453-21turbostat09:24:542
337633099447434,10cyclictest3454098-21turbostat10:49:553
337633099445438,4cyclictest3481035-21turbostat12:00:003
337633099444437,4cyclictest3390190-21turbostat07:44:593
337631899441419,19cyclictest3421730-21turbostat09:19:550
337632599440433,4cyclictest3486271-21turbostat12:19:552
337632599438431,4cyclictest3411002-21turbostat08:49:562
337633099436429,4cyclictest3386073-21turbostat07:39:553
337633099435428,4cyclictest3484499-21turbostat12:14:563
337633099435424,8cyclictest3441671-21turbostat10:14:553
337633099433426,4cyclictest3464474-21turbostat11:19:553
337633099433424,6cyclictest3434153-21turbostat09:54:553
337633099432425,4cyclictest3482760-21turbostat12:05:003
337633099431424,4cyclictest3457557-21turbostat10:55:013
337633099431422,6cyclictest3464474-21turbostat11:15:013
337633099431422,6cyclictest3464474-21turbostat11:15:003
337632599431422,6cyclictest3461012-21turbostat11:09:552
337633099429393,33cyclictest3443401-21turbostat10:19:553
337633099427420,4cyclictest3477561-21turbostat11:54:553
337633099427420,4cyclictest3391929-21turbostat07:54:553
337633099425418,4cyclictest3386073-21turbostat07:35:003
337633099421413,5cyclictest3445441-21turbostat10:24:553
337633099418411,4cyclictest3450635-21turbostat10:39:553
337633099417410,4cyclictest3412733-21turbostat08:54:553
337633099416409,4cyclictest3461012-21turbostat11:05:003
337633099415408,4cyclictest3493243-21turbostat12:35:013
337632599415408,4cyclictest3457557-21turbostat10:59:552
337632599415408,4cyclictest3426921-21turbostat09:34:552
337633099413405,5cyclictest3484499-21turbostat12:10:003
337633099409401,5cyclictest3405804-21turbostat08:30:003
337633099409401,5cyclictest3405804-21turbostat08:30:003
337633099405398,4cyclictest3382234-21turbostat07:25:003
337633099405397,5cyclictest3409268-21turbostat08:40:003
337633099403396,4cyclictest3402349-21turbostat08:20:003
337633099403376,24cyclictest3402349-21turbostat08:24:563
337633099402393,6cyclictest3491512-21turbostat12:30:003
337633099402393,6cyclictest3491512-21turbostat12:30:003
337633099398391,4cyclictest3477561-21turbostat11:50:003
337633099395387,5cyclictest3377986-21turbostat07:15:003
337633099393384,6cyclictest3395432-21turbostat08:00:013
337633099391384,4cyclictest3430693-21turbostat09:40:013
337633099390383,4cyclictest3494960-21turbostat12:40:003
337633099390383,4cyclictest3382234-21turbostat07:29:563
337633099388380,5cyclictest3398894-21turbostat08:14:543
337632599388381,4cyclictest3473780-21turbostat11:44:552
337633099381374,4cyclictest3432428-21turbostat09:45:003
337633099380373,4cyclictest3472050-21turbostat11:35:003
337632599375354,18cyclictest3384330-21turbostat07:34:552
337633099368361,4cyclictest3418232-21turbostat09:05:003
337633099366355,8cyclictest3420002-21turbostat09:10:003
337633099362356,3cyclictest3435875-21turbostat09:59:553
337633099362355,4cyclictest3488001-21turbostat12:20:003
337633099355349,3cyclictest3409268-21turbostat08:44:563
337632599350342,5cyclictest3416499-21turbostat09:04:552
337632599349342,4cyclictest3418232-21turbostat09:09:552
337633099347341,4cyclictest3434153-21turbostat09:50:003
337633099346339,4cyclictest3395432-21turbostat08:04:553
337633099346337,6cyclictest3407548-21turbostat08:35:003
337633099344337,4cyclictest3462747-21turbostat11:10:003
337631899344338,3cyclictest3447173-21turbostat10:29:550
337633099341333,5cyclictest3425200-21turbostat09:25:003
337632599336329,4cyclictest3388161-21turbostat07:44:552
337632599335328,4cyclictest3462747-21turbostat11:14:552
337632599335328,4cyclictest3462747-21turbostat11:14:552
337632599334326,5cyclictest3475810-21turbostat11:49:562
337632599331324,4cyclictest3470280-21turbostat11:34:552
337632599329322,4cyclictest3481035-21turbostat12:04:552
337633099323314,6cyclictest3459278-21turbostat11:00:003
337632599323312,8cyclictest3376189-21turbostat07:14:552
337633099314308,3cyclictest3421730-21turbostat09:15:013
337632599314307,4cyclictest3405804-21turbostat08:34:552
337632599311305,3cyclictest3432428-21turbostat09:49:552
337510922820,16sleep20-21swapper/207:05:162
337633099154141,9cyclictest3423453-21turbostat09:19:593
3375944213571,23sleep10-21swapper/107:07:001
3375913213172,17sleep00-21swapper/007:06:350
337633099123116,5cyclictest3428646-21turbostat09:35:003
3376105211872,36sleep30-21swapper/307:09:073
344602921080,4sleep20-21swapper/210:20:142
33763309910594,8cyclictest3448900-21turbostat10:30:003
33763309910594,8cyclictest3412733-21turbostat08:50:003
33763259910392,8cyclictest3404081-21turbostat08:29:562
33763259910392,8cyclictest3404081-21turbostat08:29:562
34014562760,3sleep10-21swapper/108:15:211
34456492750,6sleep20-21swapper/210:20:022
34246302750,3sleep10-21swapper/109:20:281
197399610,9rtkit-daemon1972-21rtkit-daemon09:36:221
197399590,9rtkit-daemon1972-21rtkit-daemon12:35:211
197399570,8rtkit-daemon1972-21rtkit-daemon08:00:140
3376318995416,21cyclictest3493270-21grep12:35:020
3376325995329,19cyclictest3434780-21df_inode09:50:152
337631899511,11cyclictest2093-21JS10:49:100
3376320994920,25cyclictest3384328-21turbostat.cron07:30:001
197399480,39rtkit-daemon438-20systemd-journal09:10:411
197399480,39rtkit-daemon1941-21systemd07:50:341
3376318994721,22cyclictest3387345-40lsb_release07:35:240
34259882460,7sleep126-21ksoftirqd/109:25:201
34938612450,5sleep04497-21PK-Backend12:35:150
3376325994427,12cyclictest3387362-21taskset07:35:262
3376325994415,14cyclictest34884832expr12:20:142
337632099441,18cyclictest3419803-21systemd-udevd09:08:061
337631899441,16cyclictest33783342timerandwakeup07:15:040
337631899439,14cyclictest13-21ksoftirqd/010:40:000
34309572420,8chrt4497-21PK-Backend09:40:120
3376325994224,13cyclictest3439701-40lsb_release10:02:262
3376325994224,13cyclictest3439701-40lsb_release10:02:262
337632599421,36cyclictest438-20systemd-journal07:30:002
337632599421,29cyclictest3484768-21systemd-udevd12:10:142
3376320994224,13cyclictest438-20systemd-journal07:15:311
337631899429,13cyclictest13-21ksoftirqd/008:55:020
337631899428,11cyclictest13-21ksoftirqd/007:20:200
3376325994122,15cyclictest3393726-21unin-run07:55:002
3376325994115,22cyclictest3454095-21latency_hist10:45:022
3376320994124,12cyclictest438-20systemd-journal10:21:311
3376320994112,7cyclictest33884422apt-config07:40:021
337631899418,17cyclictest13-21ksoftirqd/007:55:140
337631899418,11cyclictest13-21ksoftirqd/010:15:010
337631899417,14cyclictest13-21ksoftirqd/007:10:290
337631899416,13cyclictest13-21ksoftirqd/009:00:000
3376318994126,11cyclictest3418212-21cron09:05:010
3376318994117,7cyclictest13-21ksoftirqd/008:49:550
197399410,34rtkit-daemon4497-21PK-Backend08:31:021
3376320994022,13cyclictest3404708-21df08:25:151
3376320994022,13cyclictest3404708-21df08:25:151
3376320994022,13cyclictest26-21ksoftirqd/111:09:431
3376320994021,6cyclictest26-21ksoftirqd/109:00:011
337631899409,13cyclictest13-21ksoftirqd/007:52:180
337631899407,16cyclictest13-21ksoftirqd/008:40:290
337631899407,11cyclictest13-21ksoftirqd/009:10:130
337631899407,10cyclictest13-21ksoftirqd/007:35:000
337631899405,20cyclictest13-21ksoftirqd/012:20:270
337631899405,20cyclictest13-21ksoftirqd/011:06:090
3376318994021,7cyclictest13-21ksoftirqd/011:48:400
337631899401,15cyclictest3435584-21seq09:51:260
34855332390,8sleep14497-21PK-Backend12:10:241
34485132390,11chrt3447409-21/usr/sbin/munin10:25:292
337632599393,32cyclictest3431101-21tr09:40:152
3376325993929,6cyclictest3443411-21perf10:15:002
3376325993920,15cyclictest3391956-21unin-run07:50:002
337632599391,16cyclictest1-21systemd08:15:002
337631899398,12cyclictest13-21ksoftirqd/012:09:220
337631899397,16cyclictest13-21ksoftirqd/011:00:280
337631899396,10cyclictest13-21ksoftirqd/012:10:210
337631899395,17cyclictest13-21ksoftirqd/007:25:230
337631899394,20cyclictest13-21ksoftirqd/009:56:290
337631899393,12cyclictest0-21swapper/012:01:240
3376318993918,7cyclictest13-21ksoftirqd/008:20:280
3376318993911,10cyclictest13-21ksoftirqd/011:20:280
3376318993910,12cyclictest13-21ksoftirqd/010:04:590
3376318993910,12cyclictest13-21ksoftirqd/010:04:580
3376325993828,6cyclictest3693053-21snapd11:26:042
3376325993822,12cyclictest3381840-21uptime07:20:302
3376325993822,11cyclictest438-20systemd-journal12:05:312
3376325993820,13cyclictest3394973-21cat07:55:292
3376320993823,11cyclictest3396870-21sshd08:01:311
337632099381,16cyclictest4497-21PK-Backend11:11:281
337632099381,16cyclictest4497-21PK-Backend11:11:271
3376320993810,9cyclictest26-21ksoftirqd/109:30:131
337631899386,10cyclictest13-21ksoftirqd/011:20:010
337631899384,16cyclictest13-21ksoftirqd/011:25:300
337631899383,20cyclictest13-21ksoftirqd/008:33:280
3376318993829,6cyclictest161rcuc/010:57:110
337631899381,14cyclictest3391917-21apt07:50:000
3376318993810,15cyclictest13-21ksoftirqd/008:28:090
3376318993810,15cyclictest13-21ksoftirqd/008:28:090
197399380,30rtkit-daemon4497-21PK-Backend07:10:391
34564552370,7sleep14497-21PK-Backend10:50:151
3376325993723,9cyclictest3429312-21grep09:35:132
337632599372,31cyclictest3467407-21df_inode11:20:152
3376325993722,11cyclictest3411012-21perf08:45:002
3376325993722,11cyclictest3380206-21apt-get07:20:002
3376320993721,12cyclictest3397727-21expr08:05:131
3376320993720,13cyclictest26-21ksoftirqd/111:35:231
3376320993720,13cyclictest26-21ksoftirqd/107:24:341
337632099371,14cyclictest4497-21PK-Backend12:06:571
337632099371,13cyclictest4497-21PK-Backend11:57:371
337631899376,17cyclictest13-21ksoftirqd/010:46:550
337631899376,11cyclictest13-21ksoftirqd/009:24:410
337631899375,12cyclictest13-21ksoftirqd/007:45:000
337631899374,19cyclictest13-21ksoftirqd/012:17:250
337631899374,11cyclictest13-21ksoftirqd/010:50:290
3376318993729,6cyclictest161rcuc/008:37:570
337631899372,13cyclictest13-21ksoftirqd/010:45:020
337631899371,15cyclictest3440180-21taskset10:05:100
337631899371,14cyclictest3450615-21taskset10:34:590
337631899371,10cyclictest3428766-21apt-config09:35:020
197399370,30rtkit-daemon0-21swapper/109:58:221
337632599362,8cyclictest3464996-21apt11:15:122
3376325993622,10cyclictest34-21ksoftirqd/209:55:302
3376325993619,12cyclictest3396003-21cat08:00:152
3376320993620,11cyclictest3469092-21expr11:25:161
337631899366,16cyclictest13-21ksoftirqd/008:05:260
337631899364,16cyclictest13-21ksoftirqd/008:10:310
337631899363,19cyclictest13-21ksoftirqd/009:06:010
3376318993629,5cyclictest161rcuc/011:54:010
3376318993626,7cyclictest13-21ksoftirqd/012:27:340
3376318993626,7cyclictest13-21ksoftirqd/012:27:340
337631899362,20cyclictest13-21ksoftirqd/008:16:180
337631899361,15cyclictest3426816-21taskset09:28:520
197399360,29rtkit-daemon0-21swapper/109:52:041
197399360,29rtkit-daemon0-21swapper/108:58:461
337632599359,5cyclictest141rcu_preempt10:50:212
3376325993524,7cyclictest1137-21runrttasks09:15:132
3376325993521,11cyclictest3426922-21dump-pmu-power09:30:002
3376325993521,10cyclictest3460255-21ntp_kernel_err11:00:232
3376325993520,11cyclictest3403192-21load08:20:212
3376325993518,12cyclictest3449168-21apt10:30:132
3376325993517,14cyclictest3451639-21ntp_states10:35:232
337632099357,5cyclictest141rcu_preempt11:24:441
3376320993525,6cyclictest3449291-21cpuspeed_turbos10:30:131
3376320993522,9cyclictest3452812-21cpuspeed_turbos10:40:141
337632099351,9cyclictest4497-21PK-Backend12:31:161
337632099351,9cyclictest4497-21PK-Backend11:04:391
337632099351,9cyclictest4497-21PK-Backend10:12:271
337632099351,9cyclictest4497-21PK-Backend10:03:181
337632099351,9cyclictest4497-21PK-Backend10:03:181
337632099351,9cyclictest4497-21PK-Backend08:37:291
337632099351,9cyclictest4497-21PK-Backend07:33:261
3376320993519,12cyclictest3475446-21sshd11:41:041
3376320993518,13cyclictest26-21ksoftirqd/111:50:521
3376320993518,13cyclictest26-21ksoftirqd/111:15:171
3376320993517,6cyclictest26-21ksoftirqd/109:44:551
3376320993516,14cyclictest26-21ksoftirqd/112:00:181
3376318993518,13cyclictest976-21rs:main0
337631899351,4cyclictest4497-21PK-Backend11:30:180
337631899351,14cyclictest3446819-21runrttasks10:20:450
3376325993418,12cyclictest3473567-21sshd11:37:312
337632599341,29cyclictest0-21swapper/208:51:262
3376320993420,10cyclictest3471158-21wc11:30:221
337632099341,9cyclictest4497-21PK-Backend12:18:531
337632099341,9cyclictest4497-21PK-Backend08:12:551
3376320993419,11cyclictest3477015-21cat11:45:291
337632099341,30cyclictest4497-21PK-Backend12:21:521
337631899341,14cyclictest1137-21runrttasks09:47:460
3376325993320,8cyclictest3455071-21tr10:45:232
3376325993318,11cyclictest3478979-21seq11:51:122
3376325993317,12cyclictest3692986-21snapd08:09:042
3376325993317,11cyclictest438-20systemd-journal12:30:312
3376325993316,12cyclictest3415710-21ntpq08:55:222
337632099331,9cyclictest4497-21PK-Backend09:16:211
3376320993318,11cyclictest3447560-21tr10:25:131
3376320993318,11cyclictest3391380-21grep07:45:301
3376320993317,12cyclictest3455073-21awk10:45:231
3376320993316,13cyclictest3451228-21expr10:35:131
337632099331,29cyclictest4497-21PK-Backend12:28:091
337632099331,29cyclictest4497-21PK-Backend12:28:081
337631899336,16cyclictest13-21ksoftirqd/011:40:110
337631899336,14cyclictest13-21ksoftirqd/011:37:230
337631899334,16cyclictest13-21ksoftirqd/011:12:430
337631899334,16cyclictest13-21ksoftirqd/011:12:420
3376318993318,11cyclictest3444181-21expr10:15:130
3376325993218,10cyclictest3401563-21uname08:15:232
3376325993217,11cyclictest3490989-21wc12:25:282
3376325993217,11cyclictest3490989-21wc12:25:282
3376325993217,10cyclictest438-20systemd-journal12:00:002
3376325993216,12cyclictest3407893-21sort08:35:122
337632099327,5cyclictest141rcu_preempt08:20:161
3376320993225,4cyclictest4497-21PK-Backend10:17:061
3376320993225,4cyclictest4497-21PK-Backend08:45:471
3376320993221,7cyclictest3441788-21grep10:10:011
3376320993217,11cyclictest3410609-21timerwakeupswit08:40:301
337632099321,4cyclictest4497-21PK-Backend10:56:111
337632099321,28cyclictest4497-21PK-Backend08:51:251
3376320993117,10cyclictest3395543-21grep08:00:021
3376320993016,10cyclictest3389239-21grep07:40:231
34336002290,7sleep14497-21PK-Backend09:45:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional