You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-05 - 17:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 05, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141257021180,12sleep1251rcuc/109:55:171
162399910,47rtkit-daemon0-21swapper/307:07:563
162399860,5rtkit-daemon0-21swapper/308:34:313
162399860,5rtkit-daemon0-21swapper/307:45:023
162399850,6rtkit-daemon0-21swapper/308:36:143
162399850,6rtkit-daemon0-21swapper/308:36:143
162399850,5rtkit-daemon0-21swapper/309:24:313
162399840,5rtkit-daemon0-21swapper/310:15:043
137069227963,9sleep10-21swapper/107:05:301
14679472770,6sleep02197014-21PK-Backend12:30:320
162399710,5rtkit-daemon2197014-21PK-Backend07:18:552
162399700,4rtkit-daemon0-21swapper/207:14:232
137084527039,25sleep00-21swapper/007:07:300
137067126648,11sleep20-21swapper/207:05:152
137119599638,36cyclictest1753-21gnome-shell10:20:173
1371195996212,26cyclictest1753-21gnome-shell07:50:543
1371195996016,21cyclictest1410830-21df_inode09:50:153
1371195996013,28cyclictest1753-21gnome-shell10:41:053
1371195996013,28cyclictest1753-21gnome-shell09:42:233
1371195996013,28cyclictest1753-21gnome-shell09:42:233
13790162590,8sleep01753-21gnome-shell07:40:250
1371195995914,27cyclictest1753-21gnome-shell09:36:103
1371195995913,17cyclictest1753-21gnome-shell07:15:203
1371195995911,29cyclictest1753-21gnome-shell10:45:053
1371195995910,28cyclictest854-21avahi-daemon11:15:033
1371183995923,13cyclictest1412949-21ntpq09:55:270
162399580,9rtkit-daemon1622-21rtkit-daemon12:33:321
14463622580,5chrt0-21swapper/311:30:303
1371195995813,27cyclictest1753-21gnome-shell12:00:513
1371195995813,27cyclictest1753-21gnome-shell11:55:323
1371195995813,27cyclictest1753-21gnome-shell11:47:143
1371195995813,27cyclictest1753-21gnome-shell09:16:373
1371195995813,26cyclictest1753-21gnome-shell12:10:283
1371195995813,26cyclictest1753-21gnome-shell08:41:273
1371195995812,27cyclictest1753-21gnome-shell09:34:213
1371183995822,13cyclictest1381995-21ntp_states07:55:230
1371183995822,13cyclictest1381995-21ntp_states07:55:230
1371195995714,18cyclictest1753-21gnome-shell10:25:203
1371195995713,26cyclictest1753-21gnome-shell12:37:053
1371195995713,25cyclictest1753-21gnome-shell10:35:503
1371195995712,27cyclictest1753-21gnome-shell10:30:393
1371195995712,27cyclictest1753-21gnome-shell08:15:593
1371195995712,27cyclictest1753-21gnome-shell07:13:203
1371195995712,26cyclictest1753-21gnome-shell11:23:093
1371195995712,26cyclictest1753-21gnome-shell10:55:033
1371195995712,26cyclictest1753-21gnome-shell07:25:263
1371183995725,8cyclictest1461570-21tr12:15:150
137119599568,29cyclictest1753-21gnome-shell12:25:213
1371195995613,16cyclictest1753-21gnome-shell09:02:543
1371195995613,16cyclictest1753-21gnome-shell09:02:543
1371195995612,26cyclictest1753-21gnome-shell09:06:113
1371195995612,25cyclictest1753-21gnome-shell08:46:573
1371195995612,25cyclictest1753-21gnome-shell07:55:203
1371195995612,25cyclictest1753-21gnome-shell07:55:203
1371195995610,20cyclictest1397463-21taskset09:11:363
1371183995615,15cyclictest854-21avahi-daemon12:35:030
137119599558,29cyclictest1753-21gnome-shell09:55:293
1371195995515,17cyclictest1388014-21/usr/sbin/munin08:25:253
1371195995512,25cyclictest1753-21gnome-shell11:42:023
1371195995512,25cyclictest1753-21gnome-shell10:10:183
1371195995512,25cyclictest1753-21gnome-shell10:02:023
1371195995511,26cyclictest1753-21gnome-shell08:03:253
162399540,8rtkit-daemon1622-21rtkit-daemon09:48:071
137119599549,16cyclictest1447589-21df_inode11:35:143
137119599548,28cyclictest1378991-21ntp_states07:40:243
1371195995414,16cyclictest1753-21gnome-shell09:46:393
1371195995413,25cyclictest1753-21gnome-shell11:14:583
1371195995413,15cyclictest1753-21gnome-shell11:09:023
1371195995413,15cyclictest1753-21gnome-shell11:09:023
1371195995411,26cyclictest1753-21gnome-shell11:04:273
137119599537,22cyclictest1458369-21ntp_states12:05:273
137119599537,18cyclictest1416327-21hddtemp_smartct10:05:203
1371195995311,22cyclictest1753-21gnome-shell08:55:153
162399520,10rtkit-daemon1753-21gnome-shell11:49:202
1371195995211,24cyclictest1753-21gnome-shell07:20:293
1371195995211,23cyclictest1753-21gnome-shell11:25:023
137118399521,6cyclictest1385732-21turbostat08:15:000
162399510,5rtkit-daemon2197014-21PK-Backend07:19:300
137119599519,16cyclictest1452875-21if_enp2s011:50:213
137119599518,19cyclictest1753-21gnome-shell12:21:403
137119599517,17cyclictest1753-21gnome-shell12:16:483
1371195995113,21cyclictest860997-21snmpd08:11:153
137119599508,20cyclictest1402190-21hddtemp_smartct09:25:193
137119599508,18cyclictest1753-21gnome-shell12:30:173
1371195995014,15cyclictest1387028-21munin-run08:25:013
137119599499,15cyclictest1431943-21df_inode10:50:163
1371195994911,14cyclictest1-21systemd08:10:013
1371183994921,23cyclictest1407847-21ps09:40:280
1371183994921,23cyclictest1407847-21ps09:40:280
137119599488,16cyclictest1753-21gnome-shell08:52:233
137119599487,16cyclictest1753-21gnome-shell07:32:333
14633012470,14sleep013-21ksoftirqd/012:20:140
13978032470,7chrt1753-21gnome-shell09:15:012
1371190994621,20cyclictest1391779-21latency_hist08:45:012
137119099461,20cyclictest1431944-21taskset10:50:162
137118599461,8cyclictest0-21swapper/107:57:081
137118599461,8cyclictest0-21swapper/107:57:071
1371185994524,16cyclictest26-21ksoftirqd/112:12:221
1371185994523,17cyclictest26-21ksoftirqd/107:30:001
13840392440,7sleep22197014-21PK-Backend08:10:012
137119599449,15cyclictest1753-21gnome-shell07:38:243
1371190994432,8cyclictest1069-21runrttasks11:10:512
1371190994412,16cyclictest34-21ksoftirqd/211:30:242
162399430,31rtkit-daemon1753-21gnome-shell12:24:572
1371190994328,11cyclictest855-21dbus-daemon12:35:002
1371185994324,6cyclictest26-21ksoftirqd/111:00:111
1371185994323,15cyclictest3242255-21gdbus10:55:011
162399420,29rtkit-daemon1069-21runrttasks09:31:192
1371190994236,4cyclictest0-21swapper/212:02:142
1371190994230,9cyclictest34-21ksoftirqd/209:50:112
1371190994230,8cyclictest1753-21gnome-shell11:43:242
1371190994226,12cyclictest855-21dbus-daemon09:05:012
1371190994226,12cyclictest855-21dbus-daemon09:05:012
137118599423,36cyclictest1622-21rtkit-daemon11:42:431
137118399422,9cyclictest1753-21gnome-shell09:25:180
1371183994223,14cyclictest855-21dbus-daemon11:15:210
162399410,29rtkit-daemon2197014-21PK-Backend11:21:492
13809702410,7sleep22197014-21PK-Backend07:55:002
1371190994130,7cyclictest1753-21gnome-shell10:01:462
1371190994129,9cyclictest34-21ksoftirqd/208:55:012
1371190994129,8cyclictest1753-21gnome-shell10:55:272
1371190994129,8cyclictest1753-21gnome-shell07:59:342
1371190994129,8cyclictest1753-21gnome-shell07:59:342
137119099412,5cyclictest141rcu_preempt10:50:012
1371190994125,11cyclictest438-20systemd-journal08:55:272
137119099411,11cyclictest1438223-21cron11:10:002
137119099411,11cyclictest1438223-21cron11:10:002
137118599419,15cyclictest141rcu_preempt12:25:021
1371185994130,8cyclictest64850irq/126-i91507:15:001
1371185994127,10cyclictest1389635-21sshd08:35:271
1371185994127,10cyclictest1389635-21sshd08:35:271
1371185994121,16cyclictest26-21ksoftirqd/112:00:281
1371185994113,6cyclictest26-21ksoftirqd/109:05:011
1371185994113,6cyclictest26-21ksoftirqd/109:05:011
162399400,29rtkit-daemon2197014-21PK-Backend09:16:392
162399400,27rtkit-daemon2197014-21PK-Backend10:37:322
162399400,27rtkit-daemon2197014-21PK-Backend08:30:272
1371190994029,7cyclictest1753-21gnome-shell12:07:472
1371190994029,7cyclictest1753-21gnome-shell08:21:122
1371190994028,7cyclictest1753-21gnome-shell12:12:102
1371190994023,13cyclictest1400009-21apt09:20:102
162399390,27rtkit-daemon2197014-21PK-Backend07:42:172
14486602390,6sleep20-21swapper/211:39:552
13970222392,6sleep11397028-21ntp_states09:10:221
1371190993928,7cyclictest1753-21gnome-shell12:29:252
1371190993928,7cyclictest1753-21gnome-shell11:27:222
1371190993928,7cyclictest1753-21gnome-shell11:01:472
1371190993928,7cyclictest1753-21gnome-shell10:31:192
1371190993928,7cyclictest1753-21gnome-shell10:21:192
1371190993928,7cyclictest1753-21gnome-shell07:38:292
1371190993927,8cyclictest1753-21gnome-shell08:18:382
1371190993927,7cyclictest1753-21gnome-shell11:19:242
1371190993924,11cyclictest34-21ksoftirqd/211:51:042
137119099392,32cyclictest3242253-21nautilus12:20:012
1371190993921,13cyclictest1376682-21pool-org.gnome.07:30:002
1371190993912,23cyclictest34-21ksoftirqd/207:50:002
1371185993929,6cyclictest26-21ksoftirqd/110:16:591
1371185993927,8cyclictest1753-21gnome-shell08:11:121
137118599392,33cyclictest1753-21gnome-shell08:34:581
1371185993923,12cyclictest1-21systemd07:20:011
1371185993922,13cyclictest855-21dbus-daemon10:10:011
1371185993921,14cyclictest1379122-21python307:40:271
1371185993916,7cyclictest1461587-21cpuspeed_turbos12:15:151
1371185993910,5cyclictest26-21ksoftirqd/111:15:211
137118399392,8cyclictest1383352-21turbostat08:05:010
1371183993922,13cyclictest1415432-21gpgv10:05:010
162399380,30rtkit-daemon0-21swapper/112:27:151
162399380,26rtkit-daemon2197014-21PK-Backend10:18:082
162399380,26rtkit-daemon2197014-21PK-Backend09:36:002
1371190993827,7cyclictest1753-21gnome-shell11:55:342
1371190993827,7cyclictest1753-21gnome-shell10:41:152
1371190993827,7cyclictest1753-21gnome-shell10:27:152
1371190993827,7cyclictest1753-21gnome-shell10:07:002
1371190993827,7cyclictest1753-21gnome-shell09:56:592
1371190993827,7cyclictest1753-21gnome-shell09:48:592
1371190993827,7cyclictest1753-21gnome-shell09:44:582
1371190993827,7cyclictest1753-21gnome-shell09:44:582
1371190993827,7cyclictest1753-21gnome-shell09:27:082
1371190993827,7cyclictest1753-21gnome-shell09:09:262
1371190993827,7cyclictest1753-21gnome-shell08:46:052
1371190993827,7cyclictest1753-21gnome-shell08:01:282
1371190993827,7cyclictest1753-21gnome-shell07:32:442
137119099381,15cyclictest1753-21gnome-shell12:35:032
1371185993822,11cyclictest26-21ksoftirqd/112:35:111
1371185993819,15cyclictest1069-21runrttasks09:34:261
137118399383,31cyclictest1424389-21taskset10:30:010
1371183993827,7cyclictest1399481-21sh09:20:000
1371183993823,10cyclictest1445217-21turbostat11:30:000
1371183993821,12cyclictest438-20systemd-journal11:00:000
1371183993818,16cyclictest2962348-21rs:main0
137118399381,33cyclictest2197014-21PK-Backend12:10:400
137118399381,32cyclictest2197014-21PK-Backend10:30:140
162399370,26rtkit-daemon0-21swapper/208:28:202
162399370,25rtkit-daemon2197014-21PK-Backend08:35:452
162399370,25rtkit-daemon2197014-21PK-Backend08:35:442
162399370,23rtkit-daemon0-21swapper/208:10:392
13811752370,5sleep00-21swapper/007:55:020
1371190993726,7cyclictest1753-21gnome-shell07:21:252
1371185993727,7cyclictest1753-21gnome-shell09:25:131
1371185993726,7cyclictest1377293-21basename07:35:011
1371185993725,9cyclictest26-21ksoftirqd/107:55:011
1371185993725,8cyclictest1753-21gnome-shell11:48:311
137118599372,11cyclictest141rcu_preempt08:28:581
1371185993720,13cyclictest1439088-21memory11:10:211
1371185993720,13cyclictest1-21systemd08:05:001
1371185993718,14cyclictest1426736-21df10:35:161
137118399372,7cyclictest1753-21gnome-shell10:17:170
1371183993721,12cyclictest1446939-21cron11:35:020
1371183993720,13cyclictest1416669-21ntp_states10:05:270
1371183993720,12cyclictest1395007-21sshd09:00:270
1371183993720,12cyclictest1395007-21sshd09:00:270
1371183993719,13cyclictest438-20systemd-journal07:45:020
162399360,28rtkit-daemon1506-21systemd10:10:560
137119099361,13cyclictest2197014-21PK-Backend10:10:322
1371185993625,7cyclictest1410602-21grep09:50:131
1371185993625,6cyclictest26-21ksoftirqd/111:05:181
1371185993625,6cyclictest26-21ksoftirqd/111:05:181
1371185993625,6cyclictest26-21ksoftirqd/109:20:521
1371185993623,10cyclictest26-21ksoftirqd/110:13:011
1371185993621,11cyclictest1423133-21expr10:25:151
1371185993619,13cyclictest855-21dbus-daemon08:41:561
1371185993610,22cyclictest1429678-21date10:45:021
137118399362,8cyclictest1753-21gnome-shell08:30:220
137118399362,7cyclictest1753-21gnome-shell08:29:230
1371183993626,6cyclictest1401529-21turbostat09:25:000
1371183993625,6cyclictest14398422chrt11:13:370
137118399362,12cyclictest0-21swapper/007:10:050
1371183993619,13cyclictest1397359-21users09:10:280
137118399361,32cyclictest13-21ksoftirqd/011:42:040
1371185993525,6cyclictest26-21ksoftirqd/110:45:321
1371185993524,6cyclictest26-21ksoftirqd/111:30:121
1371185993524,6cyclictest26-21ksoftirqd/110:20:251
1371185993524,6cyclictest26-21ksoftirqd/109:15:281
1371185993524,6cyclictest26-21ksoftirqd/108:59:551
1371185993524,6cyclictest26-21ksoftirqd/108:55:021
1371185993518,13cyclictest1443510-21pool-org.gnome.11:25:011
1371183993520,11cyclictest1429276-21sshd10:41:280
1371183993518,13cyclictest1410221-21pool-org.gnome.09:50:000
1371183993517,14cyclictest1422270-21awk10:20:330
137118399351,14cyclictest2197014-21PK-Backend12:09:280
1371185993424,7cyclictest1753-21gnome-shell07:23:231
1371185993424,6cyclictest26-21ksoftirqd/112:06:181
1371185993424,6cyclictest26-21ksoftirqd/111:36:191
1371185993424,6cyclictest26-21ksoftirqd/108:47:271
1371185993424,6cyclictest26-21ksoftirqd/108:06:371
1371185993423,6cyclictest26-21ksoftirqd/110:30:211
1371185993423,6cyclictest26-21ksoftirqd/107:40:001
137118599342,28cyclictest1396045-21unin-run09:10:001
1371185993419,11cyclictest26-21ksoftirqd/108:15:101
1371185993418,12cyclictest1380054-21sshd07:46:271
1371185993418,11cyclictest438-20systemd-journal11:26:291
137118399347,6cyclictest141rcu_preempt08:53:580
1371183993425,6cyclictest181irq_work/011:05:360
1371183993425,6cyclictest181irq_work/011:05:360
137118399341,9cyclictest2197014-21PK-Backend09:37:500
1371183993419,11cyclictest1465670-21cat12:25:230
1371183993419,11cyclictest1384209-21sed08:10:020
137118399341,30cyclictest0-21swapper/010:45:090
137118399341,29cyclictest2197014-21PK-Backend07:22:490
137118399341,29cyclictest1427620-21sshd10:37:280
137118399341,13cyclictest2197014-21PK-Backend12:03:490
137118399341,12cyclictest2197014-21PK-Backend09:35:010
1371185993324,5cyclictest26-21ksoftirqd/111:55:141
1371185993318,11cyclictest1453908-21sshd11:55:001
1371185993318,11cyclictest1405425-21tr09:35:111
1371185993314,15cyclictest1408590-21cut09:45:021
1371185993314,15cyclictest1408590-21cut09:45:021
1371183993321,9cyclictest1753-21gnome-shell08:43:040
1371183993318,11cyclictest1453140-21awk11:50:250
1371183993318,11cyclictest13-21ksoftirqd/008:15:190
1371183993316,13cyclictest855-21dbus-daemon11:25:010
1371183993316,13cyclictest1455046-21swap11:55:290
137118399331,4cyclictest2197014-21PK-Backend09:50:470
137118399331,4cyclictest2197014-21PK-Backend08:57:280
137118399331,28cyclictest2197014-21PK-Backend07:38:090
1371185993223,5cyclictest26-21ksoftirqd/108:22:201
1371185993216,12cyclictest1434057-21ntpq10:55:261
137118399321,8cyclictest2197014-21PK-Backend11:01:500
1371183993217,11cyclictest1433108-21latency_hist10:55:010
137118399321,4cyclictest2197014-21PK-Backend08:47:160
137118399321,4cyclictest2197014-21PK-Backend08:23:060
1371185993120,7cyclictest1414348-21idleruntime10:00:231
1371183993120,7cyclictest1375160-21/usr/sbin/munin07:25:280
137118399311,4cyclictest2197014-21PK-Backend11:48:550
137118399311,4cyclictest2197014-21PK-Backend09:06:450
137118399311,4cyclictest2197014-21PK-Backend08:38:110
137118399311,4cyclictest2197014-21PK-Backend08:38:110
137118399301,11cyclictest2197014-21PK-Backend07:30:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional