You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 19:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 23, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
293448022290,8sleep00-21swapper/007:05:160
162399910,72rtkit-daemon0-21swapper/207:09:452
293577299732,49cyclictest1753-21gnome-shell08:13:103
293496527350,15sleep10-21swapper/107:05:261
162399730,5rtkit-daemon0-21swapper/011:59:360
162399720,5rtkit-daemon0-21swapper/012:14:240
293522127140,25sleep30-21swapper/307:05:473
162399700,9rtkit-daemon1622-21rtkit-daemon09:04:262
293577299673,37cyclictest1753-21gnome-shell09:36:153
293577299662,35cyclictest1753-21gnome-shell09:43:043
293577299662,35cyclictest1753-21gnome-shell09:33:373
293577299652,35cyclictest1753-21gnome-shell12:08:373
293577299652,35cyclictest1753-21gnome-shell11:55:323
293577299652,34cyclictest1753-21gnome-shell07:52:153
293577299652,33cyclictest1753-21gnome-shell07:45:203
293577299652,31cyclictest1753-21gnome-shell07:36:483
293577299642,37cyclictest1753-21gnome-shell11:38:543
293577299642,37cyclictest1753-21gnome-shell07:12:183
2935772996419,24cyclictest3009012-21/usr/sbin/munin10:30:303
293577299641,8cyclictest1753-21gnome-shell10:24:373
293577299641,8cyclictest1753-21gnome-shell10:24:363
293577299632,36cyclictest1753-21gnome-shell07:30:223
293577299632,35cyclictest1753-21gnome-shell11:40:383
293577299632,32cyclictest1753-21gnome-shell10:18:203
293577299622,33cyclictest1753-21gnome-shell10:45:113
293577299612,34cyclictest1753-21gnome-shell12:00:333
293577299612,33cyclictest1753-21gnome-shell10:40:523
293577299612,33cyclictest1753-21gnome-shell10:40:523
293577299612,33cyclictest1753-21gnome-shell09:46:413
293577299612,32cyclictest1753-21gnome-shell12:37:203
293577299602,34cyclictest1753-21gnome-shell11:19:353
293577299602,34cyclictest1753-21gnome-shell08:16:083
293577299602,33cyclictest1753-21gnome-shell10:58:323
293577299602,33cyclictest1753-21gnome-shell09:24:303
293577299602,33cyclictest1753-21gnome-shell09:07:043
293577299602,33cyclictest1753-21gnome-shell08:23:393
293577299592,33cyclictest1753-21gnome-shell09:50:123
293577299592,33cyclictest1753-21gnome-shell07:26:053
293577299592,33cyclictest1753-21gnome-shell07:24:183
2935772995910,44cyclictest1753-21gnome-shell10:51:133
293577299582,30cyclictest1753-21gnome-shell12:25:093
293577299582,30cyclictest0-21swapper/307:44:533
162399580,8rtkit-daemon1622-21rtkit-daemon09:00:073
293577299572,8cyclictest1753-21gnome-shell12:19:163
293576399563,12cyclictest13-21ksoftirqd/009:39:550
162399560,49rtkit-daemon34-21ksoftirqd/208:45:382
30190672540,7sleep234-21ksoftirqd/210:55:322
293577299542,30cyclictest1753-21gnome-shell08:08:483
293576399541,11cyclictest2959769-21gpg-connect-age08:15:020
162399540,46rtkit-daemon3242254-21gmain11:21:523
293577299532,7cyclictest1753-21gnome-shell10:26:193
293577299532,7cyclictest1753-21gnome-shell09:14:313
162399530,8rtkit-daemon1622-21rtkit-daemon08:34:483
162399530,8rtkit-daemon1622-21rtkit-daemon08:34:483
293576399526,31cyclictest0-21swapper/007:15:020
2935763995218,19cyclictest13-21ksoftirqd/009:30:020
162399520,8rtkit-daemon1622-21rtkit-daemon09:22:242
29991222500,25sleep22197014-21PK-Backend10:00:222
29991222500,25sleep22197014-21PK-Backend10:00:212
2935772995018,27cyclictest3031691-21seq11:32:123
2935772995015,30cyclictest2982359-21expr09:15:133
2935766995032,14cyclictest855-21dbus-daemon07:36:001
293577299492,41cyclictest1753-21gnome-shell10:00:083
293577299492,41cyclictest1753-21gnome-shell10:00:083
2935772994918,27cyclictest3000808-21cut10:05:213
2935766994915,8cyclictest2935757-21cyclictest12:19:441
293576699491,21cyclictest141rcu_preempt09:30:241
293576399497,19cyclictest141rcu_preempt09:50:020
293576399492,24cyclictest1753-21gnome-shell07:50:240
162399490,43rtkit-daemon860997-21snmpd08:50:313
29855012480,9sleep126-21ksoftirqd/109:25:141
293577299482,42cyclictest1753-21gnome-shell07:15:593
2935772994822,21cyclictest1753-21gnome-shell11:46:373
2935772994822,21cyclictest1753-21gnome-shell11:46:363
293576399485,18cyclictest13-21ksoftirqd/007:45:020
2935763994823,8cyclictest13-21ksoftirqd/011:20:170
162399480,34rtkit-daemon2197014-21PK-Backend07:16:012
293577299472,28cyclictest1753-21gnome-shell08:56:223
2935772994719,24cyclictest1753-21gnome-shell08:00:023
2935772994719,24cyclictest1753-21gnome-shell08:00:013
293577299471,41cyclictest2955466-21grep08:00:233
2935772994712,30cyclictest2971759-21kernelversion08:45:223
2935766994722,21cyclictest2966477-40lsb_release08:30:261
2935766994722,21cyclictest2966477-40lsb_release08:30:251
2935766994720,22cyclictest26-21ksoftirqd/112:30:001
2935763994735,8cyclictest13-21ksoftirqd/007:58:480
2935763994735,8cyclictest13-21ksoftirqd/007:58:480
2935763994726,14cyclictest2966019-21expr08:30:150
2935763994726,14cyclictest2966019-21expr08:30:150
2935763994724,18cyclictest3006051-21irqstats10:20:230
2935763994724,18cyclictest3006051-21irqstats10:20:220
2935763994721,9cyclictest13-21ksoftirqd/009:16:040
293576399471,29cyclictest13-21ksoftirqd/011:27:290
293576399471,17cyclictest3008756-21munin-run10:30:000
162399470,40rtkit-daemon2197014-21PK-Backend11:15:122
162399470,31rtkit-daemon1753-21gnome-shell11:05:391
30424702460,15sleep1251rcuc/112:00:311
2935766994621,21cyclictest3053369-21cron12:35:001
2935763994635,8cyclictest13-21ksoftirqd/008:36:280
2935763994623,9cyclictest13-21ksoftirqd/012:07:560
2935763994622,9cyclictest13-21ksoftirqd/012:37:270
2935763994622,11cyclictest141rcu_preempt09:00:170
293576399461,7cyclictest13-21ksoftirqd/011:55:010
293576399461,7cyclictest13-21ksoftirqd/011:45:290
293576399461,7cyclictest13-21ksoftirqd/011:45:280
293576399461,25cyclictest3005286-21sh10:20:000
162399460,33rtkit-daemon1753-21gnome-shell11:30:291
162399460,32rtkit-daemon3001789-21unattended-upgr10:09:531
162399460,32rtkit-daemon0-21swapper/110:55:301
293577299458,33cyclictest1753-21gnome-shell08:27:263
293577299452,39cyclictest1753-21gnome-shell09:28:263
293577299452,37cyclictest3045381-21hddtemp_smartct12:10:203
2935772994513,27cyclictest2179-21gvfsd-metadata08:35:033
2935770994513,7cyclictest34-21ksoftirqd/211:50:132
293576399457,17cyclictest13-21ksoftirqd/008:49:240
2935763994521,7cyclictest13-21ksoftirqd/008:08:450
2935763994514,16cyclictest13-21ksoftirqd/010:44:480
2935763994514,16cyclictest13-21ksoftirqd/010:44:470
2935763994512,12cyclictest13-21ksoftirqd/007:40:020
293576399451,19cyclictest13-21ksoftirqd/010:30:110
162399450,31rtkit-daemon2991323-21cat09:40:110
162399450,31rtkit-daemon1753-21gnome-shell11:36:051
293577299442,8cyclictest1753-21gnome-shell12:21:563
293577299442,38cyclictest1753-21gnome-shell10:38:453
2935772994419,19cyclictest42-21ksoftirqd/311:52:193
2935772994415,24cyclictest2969515-21/usr/sbin/munin08:40:303
293577299441,4cyclictest0-21swapper/310:14:003
2935766994427,13cyclictest1-21systemd09:15:011
2935766994422,19cyclictest2994542-21turbostat.cron09:49:591
2935766994414,8cyclictest26-21ksoftirqd/109:05:291
293576399448,16cyclictest13-21ksoftirqd/007:20:340
293576399443,19cyclictest13-21ksoftirqd/008:40:270
2935763994427,12cyclictest2945085-21cron07:35:000
2935763994425,5cyclictest141rcu_preempt07:25:160
2935763994424,6cyclictest13-21ksoftirqd/009:05:080
2935763994414,17cyclictest13-21ksoftirqd/007:17:370
293576399441,12cyclictest13-21ksoftirqd/012:20:120
2935763994410,19cyclictest13-21ksoftirqd/010:14:470
162399440,31rtkit-daemon1753-21gnome-shell11:42:051
162399440,30rtkit-daemon0-21swapper/109:40:261
162399440,29rtkit-daemon3035959-21apt11:45:121
162399440,29rtkit-daemon3035959-21apt11:45:111
2935772994316,23cyclictest3028866-21tr11:25:143
293577299431,38cyclictest1753-21gnome-shell11:03:093
2935770994323,7cyclictest34-21ksoftirqd/211:38:312
2935770994321,17cyclictest3036569-40lsb_release11:45:232
2935770994321,17cyclictest3036569-40lsb_release11:45:232
293576699435,24cyclictest141rcu_preempt07:15:021
2935766994325,13cyclictest3029911-40apt-check11:27:161
2935766994317,22cyclictest3013296-21which10:40:281
2935766994317,22cyclictest3013296-21which10:40:281
293576399438,17cyclictest13-21ksoftirqd/007:45:450
293576399433,11cyclictest13-21ksoftirqd/011:05:010
293576399432,7cyclictest1753-21gnome-shell11:35:000
293576399431,5cyclictest2974499-21systemd-run08:55:010
293576399431,15cyclictest13-21ksoftirqd/010:00:130
293576399431,15cyclictest13-21ksoftirqd/010:00:120
2935763994310,19cyclictest13-21ksoftirqd/011:15:310
29556172420,5sleep2331rcuc/208:00:272
2935772994213,24cyclictest3021756-21expr11:05:143
293577099429,5cyclictest141rcu_preempt09:30:252
2935766994222,12cyclictest26-21ksoftirqd/109:50:181
293576399426,21cyclictest13-21ksoftirqd/011:12:410
293576399423,9cyclictest13-21ksoftirqd/011:05:130
162399420,30rtkit-daemon2197014-21PK-Backend07:41:392
162399420,29rtkit-daemon0-21swapper/109:59:351
29474352410,7chrt0-21swapper/107:40:101
2935772994111,26cyclictest2287-21gmain10:00:003
293577099417,5cyclictest141rcu_preempt12:20:162
2935770994123,6cyclictest34-21ksoftirqd/211:12:262
2935770994123,13cyclictest34-21ksoftirqd/209:40:252
2935770994122,14cyclictest2937383-40apt-check07:12:182
293577099410,37cyclictest1622-21rtkit-daemon09:16:582
293576699417,15cyclictest141rcu_preempt10:14:591
2935766994130,7cyclictest3014744-21sed10:45:211
2935766994110,6cyclictest141rcu_preempt08:45:221
293576399419,18cyclictest13-21ksoftirqd/008:15:280
293576399417,14cyclictest0-21swapper/010:46:360
2935763994120,9cyclictest13-21ksoftirqd/010:55:120
2935763994112,9cyclictest13-21ksoftirqd/010:52:110
293576399411,17cyclictest13-21ksoftirqd/008:25:170
162399410,28rtkit-daemon13-21ksoftirqd/010:35:200
30266102400,3sleep13026611-21grep11:20:031
29630182400,7sleep22197014-21PK-Backend08:20:322
2935770994023,13cyclictest2990676-40apt-check09:36:192
2935770994021,15cyclictest34-21ksoftirqd/207:55:292
2935770994021,15cyclictest34-21ksoftirqd/207:55:292
293576699409,5cyclictest141rcu_preempt07:48:491
2935766994026,10cyclictest2970683-21sshd08:41:071
2935766994025,11cyclictest2954465-21cron08:00:011
2935766994025,11cyclictest2954465-21cron08:00:011
2935766994024,12cyclictest26-21ksoftirqd/110:30:301
2935766994024,12cyclictest1-21systemd10:40:011
2935766994021,15cyclictest3017415-40lsb_release10:51:211
2935766994021,14cyclictest26-21ksoftirqd/111:25:011
2935766994020,12cyclictest26-21ksoftirqd/107:50:131
2935766994017,19cyclictest2968915-40apt-check08:36:181
2935766994016,20cyclictest2944256-21sed07:30:221
293576699401,27cyclictest1069-21runrttasks12:13:321
293576399406,18cyclictest13-21ksoftirqd/009:51:460
293576399406,15cyclictest0-21swapper/008:59:220
293576399406,10cyclictest13-21ksoftirqd/008:03:310
293576399402,8cyclictest1753-21gnome-shell10:05:130
2935763994013,9cyclictest13-21ksoftirqd/012:16:170
293576399401,19cyclictest13-21ksoftirqd/011:35:320
2935763994010,5cyclictest141rcu_preempt09:30:140
2935763994010,15cyclictest1069-21runrttasks09:14:140
162399400,29rtkit-daemon0-21swapper/111:51:591
162399400,28rtkit-daemon3020425-21strings11:00:261
293577299391,33cyclictest3023312-21chrt11:10:133
2935770993925,10cyclictest2952477-21sshd07:51:502
293577099392,33cyclictest1753-21gnome-shell10:36:252
293577099391,5cyclictest141rcu_preempt08:30:012
293576699399,5cyclictest141rcu_preempt08:27:261
2935766993922,13cyclictest26-21ksoftirqd/109:18:271
2935766993921,14cyclictest26-21ksoftirqd/108:55:011
2935766993921,13cyclictest2998599-40apt-check10:00:191
2935766993921,13cyclictest2998599-40apt-check10:00:191
2935766993920,14cyclictest26-21ksoftirqd/111:15:011
2935766993920,14cyclictest26-21ksoftirqd/107:15:041
2935766993914,21cyclictest2958930-21awk08:10:241
293576399399,12cyclictest0-21swapper/008:20:570
293576399397,10cyclictest141rcu_preempt12:30:100
162399390,27rtkit-daemon2197014-21PK-Backend07:26:382
293577299381,14cyclictest3051876-21chrt12:30:033
293577099388,15cyclictest141rcu_preempt10:07:422
293577099385,11cyclictest34-21ksoftirqd/208:51:062
2935770993823,11cyclictest34-21ksoftirqd/210:45:112
2935770993821,13cyclictest34-21ksoftirqd/212:04:552
2935770993819,15cyclictest3035461-21pool-org.gnome.11:45:002
2935770993819,14cyclictest438-20systemd-journal10:20:012
293576699382,32cyclictest3006443-21python310:20:311
293576699382,32cyclictest3006443-21python310:20:311
2935766993821,12cyclictest884-21gdbus08:15:041
2935766993820,14cyclictest26-21ksoftirqd/112:05:151
293576699381,14cyclictest2197014-21PK-Backend11:55:401
293576399389,6cyclictest141rcu_preempt12:00:280
293576399385,16cyclictest13-21ksoftirqd/009:55:140
2935763993814,6cyclictest141rcu_preempt09:21:310
293576399381,19cyclictest3050922-21ntp_states12:25:230
162399380,30rtkit-daemon34-21ksoftirqd/209:28:332
162399380,27rtkit-daemon2197014-21PK-Backend07:38:142
162399380,25rtkit-daemon0-21swapper/112:22:481
293577099373,29cyclictest141rcu_preempt10:29:592
2935770993726,7cyclictest2943770-21cpuspeed_turbos07:30:132
293577099372,31cyclictest1753-21gnome-shell08:11:002
293577099372,12cyclictest1753-21gnome-shell08:59:362
2935770993720,12cyclictest2960818-21memory08:15:252
2935770993713,5cyclictest141rcu_preempt12:30:272
293577099371,32cyclictest331rcuc/212:29:562
2935766993728,5cyclictest2956672-21tr08:05:131
2935766993721,12cyclictest3004645-21processes10:15:301
2935766993721,12cyclictest26-21ksoftirqd/109:35:141
293576699371,7cyclictest2984623-21python309:20:281
293576399373,12cyclictest13-21ksoftirqd/011:40:210
293577099365,9cyclictest2197014-21PK-Backend11:25:522
2935770993626,7cyclictest1753-21gnome-shell12:19:492
2935770993624,7cyclictest3026839-21tr11:20:142
2935770993622,10cyclictest3039780-21/usr/sbin/munin11:55:122
293577099362,12cyclictest1753-21gnome-shell11:07:382
2935770993621,11cyclictest2980000-21cron09:10:002
2935770993612,5cyclictest141rcu_preempt07:20:252
2935766993622,10cyclictest3007309-21/usr/sbin/munin10:25:131
293576699362,11cyclictest1753-21gnome-shell12:40:031
2935766993620,5cyclictest26-21ksoftirqd/108:55:151
2935766993619,13cyclictest855-21dbus-daemon07:25:001
2935766993618,6cyclictest26-21ksoftirqd/109:00:141
2935766993610,5cyclictest141rcu_preempt07:28:051
293577099357,9cyclictest34-21ksoftirqd/210:11:212
293577099357,5cyclictest141rcu_preempt08:30:262
293577099357,5cyclictest141rcu_preempt08:30:252
2935770993524,7cyclictest3005720-21cpuspeed_turbos10:20:162
2935770993524,7cyclictest3005720-21cpuspeed_turbos10:20:152
2935770993520,11cyclictest2949349-21cat07:45:152
2935770993519,13cyclictest34-21ksoftirqd/210:40:132
2935770993519,13cyclictest34-21ksoftirqd/210:40:122
2935770993518,6cyclictest34-21ksoftirqd/210:35:022
293576699357,6cyclictest141rcu_preempt08:24:351
293577099346,9cyclictest34-21ksoftirqd/209:49:142
2935770993420,10cyclictest2957944-21/usr/sbin/munin08:10:002
2935770993417,14cyclictest2969282-21idleruntime-cro08:40:012
2935770993411,5cyclictest141rcu_preempt12:35:202
2935770993410,5cyclictest141rcu_preempt09:10:562
2935766993423,7cyclictest1753-21gnome-shell08:02:191
293577099336,9cyclictest141rcu_preempt10:51:112
293577099334,9cyclictest34-21ksoftirqd/209:50:322
293577099333,5cyclictest141rcu_preempt12:06:172
2935770993315,14cyclictest34-21ksoftirqd/211:30:502
293577099331,29cyclictest2197014-21PK-Backend08:44:092
293577099329,5cyclictest141rcu_preempt09:55:592
293577099322,12cyclictest141rcu_preempt11:00:232
293577099318,5cyclictest141rcu_preempt12:10:452
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional