You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 19:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 11, 2025 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
59520722200,8sleep00-21swapper/007:05:270
162399880,8rtkit-daemon711843-21ntp_states12:35:283
6624922840,3chrt662089-21/usr/sbin/munin10:15:161
162399820,6rtkit-daemon2197014-21PK-Backend08:18:113
6676552810,5sleep20-21swapper/210:30:152
162399810,6rtkit-daemon0-21swapper/312:34:393
162399800,6rtkit-daemon2197014-21PK-Backend08:23:153
162399800,6rtkit-daemon2197014-21PK-Backend08:23:143
6887352790,4chrt688515-21/usr/sbin/munin11:30:151
162399790,5rtkit-daemon2197014-21PK-Backend10:45:563
162399710,55rtkit-daemon0-21swapper/107:05:341
59566427040,23sleep30-21swapper/307:09:293
162399700,9rtkit-daemon597557-21munin-node10:32:000
162399700,4rtkit-daemon0-21swapper/007:25:410
162399690,5rtkit-daemon2197014-21PK-Backend07:30:030
162399680,8rtkit-daemon921-21thermald10:11:380
162399670,4rtkit-daemon2197014-21PK-Backend10:55:320
59540126639,21sleep20-21swapper/207:06:032
162399640,4rtkit-daemon0-21swapper/010:02:300
162399620,9rtkit-daemon1622-21rtkit-daemon11:53:101
162399620,9rtkit-daemon1622-21rtkit-daemon11:53:091
162399610,9rtkit-daemon1622-21rtkit-daemon10:59:491
162399580,9rtkit-daemon1622-21rtkit-daemon10:01:251
6418702560,7chrt0-21swapper/009:17:030
162399560,7rtkit-daemon1622-21rtkit-daemon10:23:371
162399550,8rtkit-daemon1622-21rtkit-daemon07:57:001
162399550,8rtkit-daemon1622-21rtkit-daemon07:56:591
59584899546,19cyclictest669893-21ntp_states10:35:272
162399540,44rtkit-daemon1506-21systemd11:13:441
59584899518,15cyclictest616479-21ps08:05:302
59584899515,24cyclictest1753-21gnome-shell12:10:142
595844995139,9cyclictest26-21ksoftirqd/109:34:491
59584899501,24cyclictest1753-21gnome-shell07:20:022
162399500,6rtkit-daemon1622-21rtkit-daemon11:07:541
6282232490,8sleep12197014-21PK-Backend08:40:181
6282232490,8sleep12197014-21PK-Backend08:40:181
59584899498,16cyclictest1753-21gnome-shell09:17:092
595852994818,25cyclictest654922-21apt09:55:143
59584899486,9cyclictest619854-21ntp_kernel_pll_08:15:252
595844994819,8cyclictest26-21ksoftirqd/107:10:261
162399480,39rtkit-daemon1506-21systemd11:00:311
6603012470,7sleep12197014-21PK-Backend10:10:131
59584899475,14cyclictest34-21ksoftirqd/211:40:222
162399470,37rtkit-daemon1506-21systemd12:00:241
595852994615,26cyclictest608717-21basename07:45:123
59584899468,13cyclictest1753-21gnome-shell08:51:432
59584899466,13cyclictest699604-21grep12:00:232
59584899464,15cyclictest1753-21gnome-shell12:06:272
59584899460,16cyclictest1753-21gnome-shell08:35:582
7082592450,16sleep126-21ksoftirqd/112:25:251
59584899459,12cyclictest1753-21gnome-shell09:40:202
59584899457,11cyclictest649655-21sed09:40:022
595848994529,11cyclictest611146-21grep07:50:242
595848994529,11cyclictest611146-21grep07:50:242
595848994510,18cyclictest1753-21gnome-shell09:02:232
595848994510,18cyclictest1753-21gnome-shell07:47:492
595842994533,8cyclictest13-21ksoftirqd/010:16:540
162399450,38rtkit-daemon1506-21systemd07:50:161
162399450,38rtkit-daemon1506-21systemd07:50:161
59584899449,10cyclictest1753-21gnome-shell12:37:342
59584899449,10cyclictest1753-21gnome-shell10:28:082
59584899449,10cyclictest1753-21gnome-shell10:28:072
59584899443,24cyclictest1753-21gnome-shell12:17:482
595842994415,17cyclictest632500-21systemd-udevd08:51:320
162399440,37rtkit-daemon1506-21systemd07:46:421
162399440,32rtkit-daemon1753-21gnome-shell09:32:373
162399440,31rtkit-daemon642911-21cstates09:20:143
6840972430,7sleep11753-21gnome-shell11:15:341
595852994324,14cyclictest662877-21ntpq10:15:273
59584899439,18cyclictest1753-21gnome-shell07:26:482
59584899439,17cyclictest1753-21gnome-shell08:30:202
59584899437,9cyclictest628224-21diskstats08:40:182
59584899437,9cyclictest628224-21diskstats08:40:182
59584899436,9cyclictest647749-21munin-run09:35:012
59584899433,23cyclictest1753-21gnome-shell10:52:262
59584899433,23cyclictest1753-21gnome-shell10:06:202
595848994325,14cyclictest855-21dbus-daemon07:15:012
595848994325,13cyclictest438-20systemd-journal07:59:552
595848994325,13cyclictest438-20systemd-journal07:59:552
59584899431,37cyclictest2289-21gdbus10:15:302
595844994323,16cyclictest26-21ksoftirqd/107:20:021
595844994322,16cyclictest26-21ksoftirqd/107:45:021
162399430,36rtkit-daemon1506-21systemd11:40:541
162399430,36rtkit-daemon1506-21systemd07:35:381
162399430,35rtkit-daemon856-21gmain10:33:151
162399430,35rtkit-daemon1506-21systemd11:35:041
6468542420,8sleep02197014-21PK-Backend09:30:250
6176532420,5sleep30-21swapper/308:10:133
595852994226,12cyclictest652948-21systemd-run09:50:013
595852994226,12cyclictest652948-21systemd-run09:50:013
595852994220,18cyclictest1753-21gnome-shell07:54:553
595852994220,18cyclictest1753-21gnome-shell07:54:553
595852994218,19cyclictest628723-21sshd08:40:313
595852994218,19cyclictest628723-21sshd08:40:313
595852994214,24cyclictest1753-21gnome-shell07:40:163
59584899429,10cyclictest690654-21diskstats11:35:212
59584899428,10cyclictest1753-21gnome-shell10:21:202
59584899427,18cyclictest682029-21cat11:10:212
59584899426,18cyclictest1753-21gnome-shell12:35:032
59584899425,14cyclictest1753-21gnome-shell07:20:232
595848994212,16cyclictest34-21ksoftirqd/211:25:532
595844994224,13cyclictest659881-21pool-org.gnome.10:10:021
595844994221,16cyclictest26-21ksoftirqd/111:55:181
162399420,35rtkit-daemon1506-21systemd09:50:441
162399420,34rtkit-daemon856-21gmain11:48:501
6341352410,5sleep02197014-21PK-Backend08:55:340
59584899418,11cyclictest1753-21gnome-shell10:02:332
59584899416,13cyclictest707974-21ls12:25:172
59584899414,13cyclictest860997-21snmpd08:23:032
59584899414,13cyclictest860997-21snmpd08:23:032
59584899412,23cyclictest1753-21gnome-shell09:24:132
595844994122,14cyclictest641904-21touch09:17:191
595844994119,16cyclictest26-21ksoftirqd/110:40:161
595842994114,5cyclictest141rcu_preempt10:37:290
595852994025,11cyclictest641915-21sshd09:17:273
595852994023,13cyclictest637567-21ntp_states09:05:273
59584899409,17cyclictest34-21ksoftirqd/211:19:082
59584899407,10cyclictest693717-21/usr/sbin/munin11:45:152
59584899404,31cyclictest617606-21cat08:10:142
59584899404,15cyclictest1753-21gnome-shell08:58:012
595848994025,11cyclictest689948-21sshd11:34:292
59584899402,16cyclictest1753-21gnome-shell09:46:132
59584899402,16cyclictest1753-21gnome-shell09:46:132
59584899400,17cyclictest1753-21gnome-shell11:50:232
59584899400,17cyclictest1753-21gnome-shell11:50:232
595844994023,6cyclictest26-21ksoftirqd/112:05:161
59584499402,34cyclictest1753-21gnome-shell08:09:001
595844994022,13cyclictest438-20systemd-journal09:00:271
59584499401,13cyclictest6341362cat08:55:351
59584299402,6cyclictest651189-21munin-run09:45:000
595842994024,12cyclictest620466-21sshd08:18:280
59584299402,34cyclictest1753-21gnome-shell11:08:240
595842994022,14cyclictest1632-21dbus-daemon11:50:010
595842994020,8cyclictest141rcu_preempt10:23:420
59584299401,14cyclictest653157-21taskset09:50:090
5982542390,27chrt2197014-21PK-Backend07:15:120
59585299395,30cyclictest1753-21gnome-shell07:19:123
595852993925,10cyclictest42-21ksoftirqd/312:11:243
59585299392,33cyclictest1753-21gnome-shell07:33:483
59585299392,32cyclictest1753-21gnome-shell09:50:103
595852993923,12cyclictest708869-21seq12:27:213
595852993923,12cyclictest625585-21systemd-udevd08:32:153
595852993922,13cyclictest696949-21awk11:55:003
595852993922,13cyclictest696949-21awk11:55:003
595852993922,12cyclictest438-20systemd-journal12:03:463
595852993922,12cyclictest438-20systemd-journal11:37:293
595852993921,13cyclictest438-20systemd-journal11:15:293
59584899399,16cyclictest34-21ksoftirqd/210:40:292
59584899399,16cyclictest34-21ksoftirqd/210:13:242
59584899397,10cyclictest678846-21awk11:00:252
59584899395,14cyclictest1069-21runrttasks08:46:502
595848993924,11cyclictest644849-21df_inode09:25:162
595848993924,11cyclictest644849-21df_inode09:25:162
595848993922,13cyclictest637547-21ntpq09:05:242
59584899392,12cyclictest607412-21smartctl07:40:192
595848993919,7cyclictest34-21ksoftirqd/211:22:222
595844993928,7cyclictest1753-21gnome-shell10:45:181
59584499392,13cyclictest1753-21gnome-shell09:28:031
59584499392,13cyclictest1753-21gnome-shell09:28:021
595844993917,6cyclictest141rcu_preempt08:51:311
595844993915,20cyclictest26-21ksoftirqd/108:40:011
595844993913,22cyclictest620610-21/usr/sbin/munin08:20:021
595842993924,11cyclictest675729-21sshd10:50:510
595842993923,12cyclictest703751-21sshd12:13:290
595842993922,6cyclictest13-21ksoftirqd/011:23:580
595842993922,12cyclictest623290-21cut08:25:250
595842993920,6cyclictest13-21ksoftirqd/011:57:020
595842993920,6cyclictest13-21ksoftirqd/009:04:370
595842993919,16cyclictest700676-21apt12:05:130
595842993911,6cyclictest141rcu_preempt12:25:120
162399390,29rtkit-daemon0-21swapper/308:27:103
162399390,28rtkit-daemon2197014-21PK-Backend12:18:283
162399390,26rtkit-daemon0-21swapper/208:28:212
595852993820,14cyclictest3242254-21gmain08:40:013
59584899388,9cyclictest672986-21diskstats10:45:192
59584899385,12cyclictest1753-21gnome-shell12:20:292
59584899383,14cyclictest1753-21gnome-shell10:56:262
59584899382,9cyclictest2197014-21PK-Backend09:12:452
595848993824,10cyclictest613935-21apt08:00:132
595848993820,13cyclictest680217-21df_inode11:05:162
595844993826,9cyclictest26-21ksoftirqd/112:25:001
595844993824,10cyclictest601182-21sshd07:21:291
595844993822,5cyclictest26-21ksoftirqd/109:05:281
59584299386,14cyclictest0-21swapper/007:23:070
59584299382,32cyclictest1753-21gnome-shell09:39:170
59584299382,32cyclictest1753-21gnome-shell08:09:510
59584299381,9cyclictest707526-21date12:25:010
59584299381,16cyclictest625683-21chrt08:33:080
162399380,8rtkit-daemon2287-21gmain07:55:210
162399380,8rtkit-daemon2287-21gmain07:55:200
162399380,26rtkit-daemon2197014-21PK-Backend09:35:193
595852993727,6cyclictest1753-21gnome-shell10:02:583
595852993722,11cyclictest42-21ksoftirqd/310:05:173
595852993721,12cyclictest614382-21cat08:00:203
595852993720,13cyclictest677093-21ntpq10:55:233
595852993720,13cyclictest42-21ksoftirqd/308:06:083
595852993719,6cyclictest42-21ksoftirqd/311:00:103
595852993717,15cyclictest438-20systemd-journal11:31:053
59584899375,10cyclictest860997-21snmpd11:58:452
595848993727,7cyclictest1069-21runrttasks07:33:082
595844993723,11cyclictest652922-21munin-plugin-st09:50:011
595844993723,11cyclictest652922-21munin-plugin-st09:50:011
595844993723,10cyclictest642850-21tr09:20:141
595844993711,21cyclictest640084-21awk09:15:001
595842993727,6cyclictest1753-21gnome-shell08:24:520
595842993727,6cyclictest1753-21gnome-shell08:24:520
59584299372,31cyclictest1753-21gnome-shell12:31:350
595842993721,12cyclictest696339-21sensors_temp11:50:290
595842993721,12cyclictest696339-21sensors_temp11:50:290
595842993721,12cyclictest640200-21date09:15:010
595842993720,13cyclictest645213-21ntpq09:25:250
595842993720,13cyclictest645213-21ntpq09:25:250
59584299371,12cyclictest0-21swapper/011:43:110
162399370,31rtkit-daemon217-1kworker/1:1H07:31:301
59585299364,11cyclictest42-21ksoftirqd/308:51:273
59585299363,29cyclictest1753-21gnome-shell07:39:283
59585299362,14cyclictest633457-21df08:55:193
595852993620,12cyclictest664793-21sshd10:20:293
595852993620,12cyclictest1-21systemd12:25:023
595852993620,11cyclictest646110-21sed09:30:023
595852993620,11cyclictest646110-21sed09:30:023
595852993619,12cyclictest468-21systemd-udevd11:28:503
595852993618,13cyclictest603392-21cat07:30:033
595852993611,6cyclictest6010962chrt07:20:303
59585299361,13cyclictest6860722timerandwakeup11:20:303
59584899368,11cyclictest34-21ksoftirqd/209:50:092
59584899365,12cyclictest34-21ksoftirqd/209:56:492
59584899362,8cyclictest34-21ksoftirqd/207:35:272
595844993627,6cyclictest1753-21gnome-shell08:33:041
595844993624,7cyclictest26-21ksoftirqd/110:40:011
595844993618,13cyclictest622510-21sed08:25:021
595844993618,13cyclictest622510-21sed08:25:021
595844993615,17cyclictest676142-21unin-run10:55:021
595842993626,7cyclictest711272-21tr12:35:160
595842993620,12cyclictest630761-21seq08:46:330
595842993620,12cyclictest618869-21/usr/sbin/munin08:15:000
595842993620,12cyclictest610008-21seq07:47:450
595842993617,14cyclictest3242253-21nautilus12:05:010
595852993524,7cyclictest596240-21cpuspeed_turbos07:10:133
595852993522,9cyclictest1753-21gnome-shell07:58:313
595852993522,9cyclictest1753-21gnome-shell07:58:313
59585299352,15cyclictest680495-21/usr/sbin/munin11:05:243
59585299351,7cyclictest2197014-21PK-Backend10:33:093
59585299351,30cyclictest2197014-21PK-Backend10:11:253
595844993524,6cyclictest26-21ksoftirqd/111:20:231
595844993524,6cyclictest26-21ksoftirqd/109:55:161
595844993524,6cyclictest26-21ksoftirqd/109:44:241
595844993524,6cyclictest26-21ksoftirqd/107:25:221
595844993523,8cyclictest1753-21gnome-shell12:13:151
595844993521,10cyclictest705487-21sshd12:18:291
595844993518,12cyclictest438-20systemd-journal08:29:591
595842993525,7cyclictest1069-21runrttasks10:45:540
595842993521,10cyclictest656156-21seq09:57:160
595842993519,12cyclictest672023-21seq10:41:070
162399350,3rtkit-daemon1622-21rtkit-daemon07:10:260
162399350,28rtkit-daemon0-21swapper/110:26:281
162399350,28rtkit-daemon0-21swapper/110:26:281
162399350,27rtkit-daemon1506-21systemd11:01:020
595852993416,14cyclictest3426826-21update-manager12:05:293
595852993411,20cyclictest391irq_work/310:29:403
595852993411,20cyclictest391irq_work/310:29:403
595844993424,6cyclictest26-21ksoftirqd/112:30:111
595844993424,6cyclictest26-21ksoftirqd/109:35:111
595844993424,6cyclictest26-21ksoftirqd/108:00:371
59584299347,12cyclictest141rcu_preempt07:40:010
59584299343,14cyclictest1753-21gnome-shell11:17:280
59584299342,10cyclictest1753-21gnome-shell11:25:060
59584299341,9cyclictest705438-21taskset12:17:500
59584299341,8cyclictest1358-1kworker/u9:111:32:290
595842993418,12cyclictest666244-21cat10:25:230
595842993418,12cyclictest666244-21cat10:25:230
59584299341,13cyclictest2197014-21PK-Backend07:40:180
59585299339,6cyclictest141rcu_preempt08:49:593
59585299338,5cyclictest141rcu_preempt12:00:003
595852993322,7cyclictest671423-21latency10:40:233
595852993322,7cyclictest411rcuc/311:42:413
595852993322,6cyclictest42-21ksoftirqd/309:43:033
59585299332,26cyclictest1753-21gnome-shell10:50:103
59585299331,8cyclictest2197014-21PK-Backend10:37:363
595852993318,11cyclictest682333-21sh11:10:273
595852993318,11cyclictest638781-21cpuspeed_turbos09:10:153
595852993317,12cyclictest694474-21ntpq11:45:273
595852993317,12cyclictest1-21systemd09:05:013
595844993323,6cyclictest711878-21sshd12:35:291
595844993322,6cyclictest26-21ksoftirqd/111:25:291
595844993322,6cyclictest26-21ksoftirqd/108:45:191
59584299336,12cyclictest141rcu_preempt10:07:110
59584299336,12cyclictest141rcu_preempt08:04:080
595842993322,7cyclictest611939-21grep07:55:000
595842993322,7cyclictest611939-21grep07:55:000
59584299332,27cyclictest0-21swapper/011:10:140
595842993321,8cyclictest628398-21grep08:40:240
595842993321,8cyclictest628398-21grep08:40:230
595842993321,7cyclictest691739-21sh11:39:590
59584299331,8cyclictest2197014-21PK-Backend09:20:040
595842993318,11cyclictest636947-21grep09:05:150
595842993317,12cyclictest626341-21expr08:35:150
59584299331,11cyclictest0-21swapper/009:45:320
59584299331,11cyclictest0-21swapper/009:45:320
595844993222,5cyclictest26-21ksoftirqd/108:10:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional