You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 21:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Tue Aug 26, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991250,31rtkit-daemon1753-21gnome-shell11:06:470
2529936999587,5cyclictest26-21ksoftirqd/109:30:291
252993699942,89cyclictest0-21swapper/109:24:361
162399940,76rtkit-daemon0-21swapper/307:07:423
252993699932,86cyclictest2197014-21PK-Backend09:26:251
252993699912,6cyclictest0-21swapper/109:17:281
252970629045,37sleep10-21swapper/107:08:451
2529941998982,5cyclictest0-21swapper/209:28:592
2529941998881,5cyclictest0-21swapper/209:22:482
25289852880,5sleep00-21swapper/007:05:240
2529936998679,4cyclictest2197014-21PK-Backend09:14:141
2529942998325,8cyclictest42-21ksoftirqd/309:15:033
2529941998375,5cyclictest2576951-21sshd09:17:282
26112832810,6sleep21753-21gnome-shell10:55:162
252994199792,72cyclictest1753-21gnome-shell09:30:392
162399750,6rtkit-daemon2197014-21PK-Backend08:24:201
162399750,6rtkit-daemon0-21swapper/108:45:241
2529941997468,4cyclictest0-21swapper/209:14:382
162399740,5rtkit-daemon0-21swapper/109:47:341
162399740,52rtkit-daemon2197014-21PK-Backend07:59:110
162399730,6rtkit-daemon2197014-21PK-Backend08:25:091
162399720,5rtkit-daemon0-21swapper/110:08:431
162399720,5rtkit-daemon0-21swapper/108:04:571
25651972710,7sleep21753-21gnome-shell08:45:152
2529933997146,9cyclictest13-21ksoftirqd/009:30:440
2529933997021,45cyclictest1753-21gnome-shell09:19:210
2529933997020,45cyclictest1753-21gnome-shell09:24:580
252994299682,62cyclictest1753-21gnome-shell09:20:383
2529933996818,40cyclictest1753-21gnome-shell09:28:480
252962726752,9sleep20-21swapper/207:07:572
252994199661,39cyclictest2556424-21/usr/sbin/munin08:20:302
162399650,9rtkit-daemon1753-21gnome-shell07:51:140
25989072630,7sleep12197014-21PK-Backend10:20:211
252994299632,56cyclictest1753-21gnome-shell09:26:523
2529941996316,25cyclictest2594984-21cpuspeed10:10:152
2529941996316,25cyclictest2594984-21cpuspeed10:10:152
25698312620,8chrt2197014-21PK-Backend09:00:013
252994199622,42cyclictest1753-21gnome-shell08:36:092
252993399622,47cyclictest1753-21gnome-shell09:14:090
252994199612,40cyclictest1753-21gnome-shell12:28:142
252994199611,36cyclictest2639069-21taskset12:11:362
252994199611,36cyclictest2567450-21taskset08:50:312
26085052600,6sleep2331rcuc/210:45:342
252994199602,41cyclictest860997-21snmpd10:44:122
252994199602,30cyclictest1753-21gnome-shell11:20:192
162399600,5rtkit-daemon860997-21snmpd07:28:163
252994299592,53cyclictest1753-21gnome-shell09:13:493
252994199593,38cyclictest1753-21gnome-shell10:54:082
252994199592,30cyclictest1753-21gnome-shell07:22:392
252994199591,29cyclictest2618362-21df11:15:172
162399590,9rtkit-daemon1622-21rtkit-daemon10:34:262
252994199583,31cyclictest1753-21gnome-shell11:53:092
252994199582,31cyclictest1753-21gnome-shell09:02:222
252993699586,47cyclictest2602309-21diskstats10:30:231
252994199572,30cyclictest1753-21gnome-shell07:36:042
252994199572,29cyclictest1753-21gnome-shell08:15:062
2529942995645,7cyclictest1069-21runrttasks09:30:213
252994199562,31cyclictest1753-21gnome-shell07:34:022
252994199562,29cyclictest1753-21gnome-shell10:27:032
252994199562,29cyclictest1753-21gnome-shell09:43:102
252994199561,31cyclictest860997-21snmpd11:43:272
252994199561,31cyclictest860997-21snmpd09:57:592
252994199561,30cyclictest2586268-21cpuspeed09:45:142
2529936995617,33cyclictest438-20systemd-journal11:29:551
25399272550,6chrt0-21swapper/107:35:151
252994199554,28cyclictest1753-21gnome-shell10:02:432
252994199552,37cyclictest1753-21gnome-shell12:22:212
252994199552,37cyclictest1753-21gnome-shell10:38:202
252994199552,36cyclictest1753-21gnome-shell11:39:282
252994199552,30cyclictest1753-21gnome-shell11:46:112
252994199552,30cyclictest1753-21gnome-shell07:28:122
252994199552,28cyclictest1753-21gnome-shell08:12:262
252994199551,27cyclictest2639925-21ls12:15:152
252994199551,27cyclictest2639925-21ls12:15:152
252994199542,37cyclictest1753-21gnome-shell08:43:322
252994199542,36cyclictest1753-21gnome-shell07:18:512
252994199542,26cyclictest2568718-21hddtemp_smartct08:55:202
2529936995417,33cyclictest2644383-21sshd12:27:281
252994199532,29cyclictest1753-21gnome-shell09:51:012
2529936995317,32cyclictest2624400-21sshd11:31:171
162399530,8rtkit-daemon1753-21gnome-shell08:47:480
252994199522,29cyclictest1753-21gnome-shell12:06:512
252994199522,29cyclictest1753-21gnome-shell11:09:532
252994199522,29cyclictest1753-21gnome-shell08:09:262
2529941995217,30cyclictest2543878-21cat07:45:232
252994199521,30cyclictest860997-21snmpd11:04:442
252994199521,29cyclictest860997-21snmpd12:38:262
252994199521,29cyclictest860997-21snmpd12:38:252
2529936995223,25cyclictest2551735-21unin-run08:10:001
252994299510,47cyclictest1622-21rtkit-daemon07:22:433
252994199512,27cyclictest1753-21gnome-shell07:50:412
252994199512,26cyclictest1753-21gnome-shell08:03:432
2529941995117,30cyclictest1632-21dbus-daemon10:20:002
2529936995119,28cyclictest2541700-21cstates07:40:141
2529936995118,28cyclictest2604167-21kernelversion10:35:271
2529936995117,30cyclictest1753-21gnome-shell11:00:181
252994199502,43cyclictest2181-21gdbus11:15:012
252994199501,26cyclictest0-21swapper/208:25:262
252994199500,46cyclictest1622-21rtkit-daemon07:13:152
2529936995022,23cyclictest2634772-40lsb_release12:00:201
2529936995020,25cyclictest2607890-21if_enp2s010:45:221
2529936995018,27cyclictest2554592-21sensors08:15:311
252993699501,12cyclictest2605479-21sort10:40:141
162399500,33rtkit-daemon2197014-21PK-Backend10:07:060
2529941994923,21cyclictest2559769-40lsb_release08:30:232
252993699492,22cyclictest2641362-21/usr/sbin/munin12:20:271
2529936994918,26cyclictest2584834-21forks09:40:191
2529936994917,7cyclictest2572579-21open_files09:05:311
2529936994914,6cyclictest0-21swapper/107:50:011
2529936994912,13cyclictest2545452-21cat07:50:201
162399490,10rtkit-daemon1753-21gnome-shell08:00:190
2529936994824,20cyclictest2595427-21kernelversion10:10:231
2529936994824,20cyclictest2595427-21kernelversion10:10:221
2529936994815,12cyclictest2590442-21ntpq09:55:291
2529936994812,10cyclictest2627012-21expr11:40:151
252993699481,11cyclictest2612670-21latency_hist11:00:021
2529936994811,11cyclictest2647805-21sshd12:36:291
2529936994811,11cyclictest2647805-21sshd12:36:281
2529936994810,10cyclictest2644939-21/usr/sbin/munin12:30:181
2529933994840,4cyclictest2197014-21PK-Backend08:42:470
162399480,33rtkit-daemon1753-21gnome-shell10:20:120
162399480,31rtkit-daemon2197014-21PK-Backend11:07:151
252994199479,34cyclictest1753-21gnome-shell09:08:152
2529941994712,30cyclictest2623734-21cut11:30:232
252993699479,10cyclictest2561903-21swap08:35:301
2529936994718,26cyclictest2547226-21kernelversion07:55:231
2529936994717,13cyclictest0-21swapper/111:56:581
2529936994713,13cyclictest2563244-21seq08:40:211
2529936994711,8cyclictest0-21swapper/109:05:021
2529936994711,11cyclictest2591849-21basename10:00:211
2529936994710,11cyclictest2589570-21taskset09:55:011
2529942994625,17cyclictest1753-21gnome-shell10:43:033
2529941994612,29cyclictest2598436-21grep10:20:142
2529936994619,23cyclictest2628667-21cpuspeed_turbos11:45:181
252993699461,19cyclictest2597845-21chrt10:16:271
252993699461,15cyclictest438-20systemd-journal10:53:441
26371212450,13sleep126-21ksoftirqd/112:05:301
252993699458,16cyclictest1753-21gnome-shell08:13:431
252993699453,18cyclictest2601669-21munin-run10:30:001
252993699452,16cyclictest1753-21gnome-shell07:27:211
252993699452,12cyclictest2626694-21sed11:40:031
2529936994515,7cyclictest2538690-21sh07:30:261
2529936994513,6cyclictest2641143-21sh12:20:011
2529936994513,6cyclictest2641143-21sh12:20:001
252993699451,21cyclictest2568149-21date08:55:011
252993699451,21cyclictest2560681-21seq08:34:181
252993399451,12cyclictest2197014-21PK-Backend08:15:020
2529942994424,15cyclictest42-21ksoftirqd/308:35:213
2529941994418,22cyclictest1069-21runrttasks12:04:412
252993699449,12cyclictest2629974-21apt-get11:50:031
252993699447,10cyclictest2620852-21seq11:20:371
2529936994413,7cyclictest2534193-21cpuspeed07:20:161
2529936994411,8cyclictest0-21swapper/111:12:311
252993399444,10cyclictest0-21swapper/009:00:410
252994199439,30cyclictest1753-21gnome-shell12:31:452
252994199432,36cyclictest2633707-21systemd-udevd11:59:122
252994199431,38cyclictest1753-21gnome-shell10:06:272
2529941994311,28cyclictest2583036-21cat09:35:152
2529936994318,21cyclictest2619529-21latency_hist11:20:021
2529936994312,7cyclictest2583462-21open_files09:35:281
2529933994331,8cyclictest1753-21gnome-shell10:10:130
2529933994331,8cyclictest1753-21gnome-shell10:10:130
26241962420,8sleep01753-21gnome-shell11:30:320
2529942994214,24cyclictest0-21swapper/312:00:213
252994199422,36cyclictest2622555-21cat11:25:312
2529941994213,24cyclictest2547461-21grep07:55:262
2529936994210,8cyclictest0-21swapper/107:10:321
162399420,30rtkit-daemon2197014-21PK-Backend07:31:370
162399420,26rtkit-daemon2612692-21/usr/sbin/munin11:00:350
26432272411,8sleep02643413-21expr12:25:130
255845924116,6sleep31753-21gnome-shell08:27:223
2529942994130,7cyclictest1069-21runrttasks08:04:043
2529942994124,13cyclictest855-21dbus-daemon07:31:563
252994299411,8cyclictest2197014-21PK-Backend12:34:333
2529942994114,8cyclictest1753-21gnome-shell11:38:463
252994199411,37cyclictest2541797-21ls07:40:152
252993699412,15cyclictest1753-21gnome-shell07:16:141
252993699412,13cyclictest2568320-21apt08:55:131
2529936994110,8cyclictest0-21swapper/112:10:201
2529942994025,11cyclictest42-21ksoftirqd/308:45:113
252994299402,34cyclictest1753-21gnome-shell07:11:283
2529942994023,13cyclictest2559648-21smartctl08:30:213
2529933994024,11cyclictest2587655-21cron09:50:000
162399400,27rtkit-daemon2197014-21PK-Backend12:22:500
252994299399,17cyclictest42-21ksoftirqd/308:50:233
2529942993926,9cyclictest1753-21gnome-shell12:15:183
2529942993926,9cyclictest1753-21gnome-shell12:15:183
252994299392,33cyclictest1753-21gnome-shell08:40:233
2529942993914,7cyclictest1753-21gnome-shell10:03:523
2529933993927,8cyclictest1069-21runrttasks08:34:310
252993399392,32cyclictest2647377-21ntp_states12:35:260
252993399392,32cyclictest2647377-21ntp_states12:35:260
2529933993922,6cyclictest13-21ksoftirqd/011:15:350
162399390,27rtkit-daemon2197014-21PK-Backend12:07:110
162399390,26rtkit-daemon2197014-21PK-Backend08:58:030
26406772380,12sleep02529926-21cyclictest12:15:310
26406772380,12sleep02529926-21cyclictest12:15:310
2529942993827,7cyclictest2553453-21sh08:15:003
2529942993824,9cyclictest1753-21gnome-shell09:49:233
2529942993823,11cyclictest2547981-21sshd07:57:283
2529942993822,12cyclictest2551542-21sshd08:08:143
2529942993820,6cyclictest42-21ksoftirqd/309:54:443
2529933993830,4cyclictest2197014-21PK-Backend07:20:190
2529933993827,7cyclictest1753-21gnome-shell10:04:460
2529933993826,8cyclictest2597354-21strings10:15:270
2529933993821,12cyclictest2561346-21expr08:35:150
2529933993813,5cyclictest141rcu_preempt10:43:560
252993399381,34cyclictest0-21swapper/007:15:150
162399380,30rtkit-daemon2197014-21PK-Backend08:22:580
162399380,29rtkit-daemon1753-21gnome-shell08:50:380
162399380,27rtkit-daemon2197014-21PK-Backend08:18:440
162399380,26rtkit-daemon2197014-21PK-Backend12:12:580
162399380,26rtkit-daemon2197014-21PK-Backend09:35:150
162399380,26rtkit-daemon2197014-21PK-Backend08:27:100
162399380,26rtkit-daemon0-21swapper/007:36:310
2529942993726,7cyclictest2556058-21memory08:20:233
2529942993724,9cyclictest1069-21runrttasks08:19:013
252994299372,32cyclictest1753-21gnome-shell11:04:003
2529942993722,10cyclictest42-21ksoftirqd/311:40:163
252993399372,7cyclictest1753-21gnome-shell08:05:320
2529933993726,7cyclictest1753-21gnome-shell10:51:160
252993399372,30cyclictest1753-21gnome-shell07:40:200
2529933993721,12cyclictest855-21dbus-daemon11:59:120
2529933993719,13cyclictest2616620-21df_inode11:10:180
252993399371,12cyclictest0-21swapper/010:45:250
162399370,24rtkit-daemon0-21swapper/010:55:280
252994299363,12cyclictest141rcu_preempt10:20:003
2529942993624,8cyclictest2599299-21python310:20:293
2529942993620,12cyclictest42-21ksoftirqd/312:10:263
2529942993620,12cyclictest42-21ksoftirqd/310:35:403
2529942993620,11cyclictest438-20systemd-journal07:53:283
2529942993619,13cyclictest3242255-21gdbus11:20:013
2529942993619,13cyclictest2628490-21awk11:45:163
2529942993619,12cyclictest438-20systemd-journal11:33:283
2529942993619,12cyclictest438-20systemd-journal07:46:013
2529942993619,12cyclictest2615355-21awk11:05:283
2529942993615,16cyclictest2633705-21systemd-udevd11:59:133
252994299361,16cyclictest2594632-21chrt10:09:533
2529942993611,20cyclictest2584051-21sshd09:38:373
252993399362,5cyclictest0-21swapper/009:56:000
2529933993620,12cyclictest2530855-21awk07:10:240
2529933993617,14cyclictest855-21dbus-daemon12:35:010
2529933993610,7cyclictest26294692sed11:45:350
252994299357,5cyclictest141rcu_preempt07:41:323
252994299353,14cyclictest1753-21gnome-shell07:38:063
2529942993524,7cyclictest1753-21gnome-shell11:24:583
2529942993523,8cyclictest1753-21gnome-shell07:18:543
2529942993522,9cyclictest1753-21gnome-shell09:41:483
2529942993522,9cyclictest1753-21gnome-shell09:09:033
2529942993520,11cyclictest42-21ksoftirqd/311:50:313
2529942993519,12cyclictest2606910-21apt-get10:45:023
2529942993518,13cyclictest2637150-21sed12:05:293
252993399358,5cyclictest141rcu_preempt11:35:180
252993399351,30cyclictest1753-21gnome-shell09:08:290
252994299341,9cyclictest2197014-21PK-Backend10:29:563
2529942993419,11cyclictest2616560-21cat11:10:163
2529942993416,13cyclictest42-21ksoftirqd/312:20:333
252994299341,13cyclictest2621977-21irqstats11:25:213
252993399347,5cyclictest141rcu_preempt12:00:430
252993399343,11cyclictest141rcu_preempt10:35:050
2529933993425,6cyclictest0-21swapper/009:53:300
2529933993423,7cyclictest1069-21runrttasks10:29:420
2529933993422,8cyclictest2536269-21tr07:25:140
2529933993419,9cyclictest13-21ksoftirqd/011:29:180
2529933993418,12cyclictest1753-21gnome-shell07:48:160
2529933993417,13cyclictest2602682-21ntpq10:30:320
252993399341,12cyclictest2197014-21PK-Backend11:40:150
252994299338,21cyclictest2571553-21turbostat09:05:003
252994299331,9cyclictest2197014-21PK-Backend10:33:183
252994299331,9cyclictest2197014-21PK-Backend09:59:333
2529942993319,11cyclictest42-21ksoftirqd/312:29:103
2529933993321,8cyclictest2584420-21/usr/sbin/munin09:40:290
26100952320,7chrt2197014-21PK-Backend10:50:303
252994299321,28cyclictest2197014-21PK-Backend12:36:533
252994299321,28cyclictest2197014-21PK-Backend12:36:533
2529933993218,10cyclictest2630785-21expr11:50:150
2529942993119,8cyclictest1753-21gnome-shell10:55:263
252993399314,5cyclictest141rcu_preempt11:20:220
252994299301,10cyclictest2197014-21PK-Backend10:10:273
252994299301,10cyclictest2197014-21PK-Backend10:10:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional