You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 16:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 17, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
341957721260,7sleep00-21swapper/019:05:280
342003429176,9sleep20-21swapper/219:09:372
35323892810,7sleep13532391-21latency_hist00:25:021
34379552810,8sleep126-21ksoftirqd/119:55:231
341977627544,22sleep30-21swapper/319:06:203
342034299748,61cyclictest3242253-21nautilus20:40:020
342034299748,61cyclictest3242253-21nautilus20:40:020
342034799702,46cyclictest3447565-21munin-plugin-st20:25:011
342034799702,46cyclictest3447565-21munin-plugin-st20:25:011
341978926939,23sleep10-21swapper/119:06:311
342034299682,33cyclictest1820-21gnome-shell22:46:180
342034299682,33cyclictest1820-21gnome-shell22:46:170
342034799671,48cyclictest3481866-21cpuspeed_turbos22:00:171
342034799671,48cyclictest3481866-21cpuspeed_turbos22:00:161
342034299671,30cyclictest3440200-21runrttasks20:00:350
3420347996614,47cyclictest3425915-21needreboot19:20:291
342034299662,45cyclictest1753-21gnome-shell19:28:170
342034299643,32cyclictest3471452-21/usr/sbin/munin21:30:230
3420347996316,42cyclictest3436144-21if_enp2s019:50:221
3420347996315,43cyclictest3445253-21switchtime20:15:311
342034299632,33cyclictest1753-21gnome-shell21:02:120
342034299632,32cyclictest1753-21gnome-shell20:11:470
162399630,9rtkit-daemon1622-21rtkit-daemon00:37:263
162399630,9rtkit-daemon1622-21rtkit-daemon00:37:263
342034799621,38cyclictest3424426-21taskset19:16:391
342034799621,37cyclictest3440420-21taskset20:02:341
342034299622,35cyclictest1753-21gnome-shell22:55:490
342034299621,32cyclictest3532631-21/usr/sbin/munin00:25:320
162399620,8rtkit-daemon1622-21rtkit-daemon23:51:413
162399620,8rtkit-daemon1622-21rtkit-daemon23:51:413
342034799612,35cyclictest1753-21gnome-shell19:30:011
342034299612,33cyclictest1753-21gnome-shell00:15:500
342034299612,32cyclictest1753-21gnome-shell00:12:320
3420347996023,32cyclictest3534150-21cron00:29:591
342034299602,30cyclictest1753-21gnome-shell22:43:570
342034299601,33cyclictest3495450-21cat22:40:020
342034299601,33cyclictest1753-21gnome-shell19:35:010
342034299601,29cyclictest3425165-21gpgconf19:20:030
162399600,46rtkit-daemon34-21ksoftirqd/223:17:232
342034799592,51cyclictest3471864-31apt-check21:30:311
342034799591,35cyclictest3525168-21taskset00:04:551
342034799591,28cyclictest0-21swapper/121:20:001
342034299591,33cyclictest3487527-21memory22:15:230
342034299591,33cyclictest3430472-21/usr/sbin/munin19:35:210
342034299591,33cyclictest3430472-21/usr/sbin/munin19:35:210
162399590,8rtkit-daemon1622-21rtkit-daemon23:58:552
342034799586,49cyclictest1753-21gnome-shell21:25:011
342034799582,51cyclictest1753-21gnome-shell19:50:021
3420347995821,32cyclictest3420520-21gpgv19:10:031
342034799581,34cyclictest3464240-21ps21:10:301
342034799581,33cyclictest3518190-21seq23:44:431
342034799581,33cyclictest3484622-21taskset22:06:101
162399580,9rtkit-daemon1622-21rtkit-daemon23:49:352
162399580,9rtkit-daemon1622-21rtkit-daemon23:49:352
34974162570,4sleep20-21swapper/222:45:132
34974162570,4sleep20-21swapper/222:45:132
342034799571,33cyclictest3535627-21taskset00:31:481
342034299572,32cyclictest3446972-21python320:20:310
342034299572,32cyclictest3446972-21python320:20:310
342034299572,31cyclictest1753-21gnome-shell22:52:190
342034299571,32cyclictest3436493-21/usr/sbin/munin19:50:290
342034299571,29cyclictest3524438-21/usr/sbin/munin00:00:260
342034299571,29cyclictest3453668-21cat20:40:250
162399570,8rtkit-daemon1622-21rtkit-daemon00:31:333
34706812560,5sleep22197014-21PK-Backend21:30:032
342034799562,30cyclictest3505944-21sort23:10:021
342034799562,30cyclictest3505944-21sort23:10:021
3420347995615,35cyclictest3460731-21awk21:00:291
342034299566,44cyclictest3513305-21seq23:30:150
342034299562,48cyclictest438-20systemd-journal23:12:290
342034299562,47cyclictest1753-21gnome-shell19:55:140
342034299562,30cyclictest3482063-21df22:00:180
342034299562,30cyclictest3482063-21df22:00:180
342034299562,29cyclictest1753-21gnome-shell23:42:100
342034299562,29cyclictest1753-21gnome-shell23:02:440
342034299562,27cyclictest3470152-40lsb_release21:27:190
342034299561,28cyclictest3526617-21chrt00:06:170
162399560,8rtkit-daemon1622-21rtkit-daemon00:21:112
3420353995531,20cyclictest1753-21gnome-shell23:35:183
3420347995520,30cyclictest3473151-21cpuspeed_turbos21:35:171
3420347995517,34cyclictest3242255-21gdbus21:45:011
3420347995517,34cyclictest3242255-21gdbus21:45:011
3420347995517,33cyclictest3486599-21seq22:14:161
3420347995516,34cyclictest3496633-21wc22:40:321
3420347995515,35cyclictest3528346-21seq00:11:171
342034299552,29cyclictest3475130-21df21:40:170
342034299552,29cyclictest3475130-21df21:40:170
342034299551,30cyclictest3518511-21grep23:45:180
342034299551,30cyclictest3518511-21grep23:45:170
342034799542,34cyclictest3508847-21ntp_states23:15:291
3420347995420,29cyclictest3512047-21irqstats23:25:231
3420347995419,31cyclictest875-21polkitd22:46:181
3420347995419,31cyclictest875-21polkitd22:46:171
3420347995416,33cyclictest3489745-21cat22:20:311
3420347995415,34cyclictest3433003-21/usr/sbin/munin19:40:321
3420347995414,35cyclictest3490925-21cut22:25:221
3420347995414,35cyclictest3490925-21cut22:25:211
3420347995413,35cyclictest3430116-21sshd19:33:471
342034299542,48cyclictest1-21systemd23:20:010
342034299542,47cyclictest1738-21gdbus23:51:560
342034299542,47cyclictest1738-21gdbus23:51:560
342034299541,29cyclictest3483688-21cstates22:05:180
3420353995329,20cyclictest1753-21gnome-shell21:45:283
3420353995321,18cyclictest42-21ksoftirqd/322:46:183
3420353995321,18cyclictest42-21ksoftirqd/322:46:183
3420353995318,12cyclictest42-21ksoftirqd/323:00:003
342034799532,46cyclictest3513780-21/usr/sbin/munin23:30:241
342034799532,46cyclictest3448787-21/usr/sbin/munin20:25:341
3420347995320,28cyclictest3503259-21needreboot23:00:231
3420347995316,32cyclictest3536922-21ntp_states00:35:301
3420347995316,32cyclictest3536922-21ntp_states00:35:301
3420347995315,33cyclictest3454874-21cat20:45:141
342034799531,47cyclictest3522144-21cpuspeed_turbos23:55:161
342034799531,46cyclictest3457227-21tr20:50:271
3420347995314,34cyclictest3487256-21expr22:15:151
342034799531,36cyclictest3520295-21cpuspeed23:50:141
342034799531,36cyclictest3520295-21cpuspeed23:50:141
3420347995313,34cyclictest3440972-21cat20:05:171
342034799531,29cyclictest707415-21gmain20:11:391
342034299532,45cyclictest1753-21gnome-shell21:00:010
3420342995321,27cyclictest3423038-21load19:15:070
3420353995226,15cyclictest42-21ksoftirqd/300:11:153
3420351995210,19cyclictest34-21ksoftirqd/219:46:482
3420347995215,33cyclictest855-21dbus-daemon22:35:011
342034799521,47cyclictest3515897-21grep23:35:291
342034799521,46cyclictest34943642sleep122:35:221
342034799521,46cyclictest3477229-21/usr/sbin/munin21:45:271
342034799521,36cyclictest3526852-21taskset00:09:171
342034799521,26cyclictest0-21swapper/123:11:491
3420342995225,22cyclictest1753-21gnome-shell22:13:520
342034299522,44cyclictest3455269-21sed20:45:220
342034299521,27cyclictest0-21swapper/022:20:240
34589282510,5chrt3458210-21/usr/sbin/munin20:55:251
342034799512,44cyclictest3462082-21diskstats21:05:191
162399510,38rtkit-daemon2197014-21PK-Backend19:15:242
162399510,36rtkit-daemon2197014-21PK-Backend20:53:153
342035399502,13cyclictest42-21ksoftirqd/320:40:163
3420353995018,13cyclictest42-21ksoftirqd/319:19:103
3420353995016,19cyclictest42-21ksoftirqd/323:42:253
3420353995016,19cyclictest42-21ksoftirqd/320:11:383
3420353995013,20cyclictest42-21ksoftirqd/319:15:033
342034799502,44cyclictest1753-21gnome-shell22:53:011
3420347995014,30cyclictest3479161-21switchtime21:50:311
342034799501,34cyclictest3454206-21taskset20:41:301
342034799501,32cyclictest3450655-21vmstat20:30:331
3420347995012,34cyclictest3469111-21cat21:25:161
3420342995022,24cyclictest3513069-21latency_hist23:30:010
342034299501,27cyclictest0-21swapper/019:11:100
162399500,38rtkit-daemon2197014-21PK-Backend19:51:012
162399500,36rtkit-daemon2197014-21PK-Backend20:37:212
162399500,36rtkit-daemon2197014-21PK-Backend20:37:212
34797222490,4sleep00-21swapper/021:55:000
342034799497,39cyclictest1358-1kworker/u9:121:59:451
3420347994915,30cyclictest3502414-21cut23:00:021
3420347994910,34cyclictest3451506-21expr20:35:161
3420347994910,34cyclictest3451506-21expr20:35:161
342034299492,43cyclictest3441060-21tr20:05:170
342034299492,42cyclictest1753-21gnome-shell21:15:180
3420342994913,32cyclictest3534659-21expr00:30:180
162399490,6rtkit-daemon1622-21rtkit-daemon00:30:302
34211202480,8chrt3421121-21if_enp2s019:10:252
3420353994813,13cyclictest42-21ksoftirqd/323:10:023
3420353994813,13cyclictest42-21ksoftirqd/323:10:013
3420353994811,20cyclictest42-21ksoftirqd/300:25:033
3420347994816,28cyclictest3530395-21latency_hist00:20:011
342034799481,43cyclictest3509805-21/usr/sbin/munin23:20:181
3420342994817,26cyclictest3530877-21cstates00:20:190
342034299481,42cyclictest3435457-21unin-run19:50:010
34678022470,5sleep3411rcuc/321:20:343
34614412470,8chrt1753-21gnome-shell21:05:003
342035399472,34cyclictest1753-21gnome-shell23:29:533
3420353994713,19cyclictest42-21ksoftirqd/319:24:453
3420347994714,29cyclictest3431145-21needreboot19:35:251
3420347994714,29cyclictest3431145-21needreboot19:35:241
3420347994710,33cyclictest3518642-21expr23:45:191
3420347994710,33cyclictest3518642-21expr23:45:191
342034299479,34cyclictest1753-21gnome-shell00:38:410
342034299479,34cyclictest1753-21gnome-shell00:38:410
3420342994720,22cyclictest1069-21runrttasks21:08:440
3420342994716,26cyclictest3522103-21cpuspeed_turbos23:55:160
3420342994712,30cyclictest438-20systemd-journal23:20:020
342035399468,14cyclictest42-21ksoftirqd/321:25:173
3420353994624,10cyclictest42-21ksoftirqd/319:55:243
342035399462,15cyclictest42-21ksoftirqd/323:12:173
3420342994618,23cyclictest1069-21runrttasks19:44:230
3420342994615,26cyclictest3516668-21latency_hist23:40:020
3420342994614,27cyclictest3449468-21date20:30:020
342034299461,40cyclictest1753-21gnome-shell21:21:520
162399460,35rtkit-daemon2197014-21PK-Backend21:00:502
162399460,34rtkit-daemon2197014-21PK-Backend23:01:052
162399460,33rtkit-daemon2197014-21PK-Backend19:42:442
162399460,32rtkit-daemon3507546-40apt-check23:12:182
162399460,32rtkit-daemon3474139-40apt-check21:36:162
162399460,32rtkit-daemon1753-21gnome-shell22:10:432
162399460,32rtkit-daemon1753-21gnome-shell21:20:392
34966292450,7sleep32197014-21PK-Backend22:40:323
34280482450,7sleep32197014-21PK-Backend19:25:313
342035399455,15cyclictest42-21ksoftirqd/320:00:153
3420353994536,6cyclictest42-21ksoftirqd/319:39:383
3420353994536,6cyclictest42-21ksoftirqd/319:39:383
3420353994522,19cyclictest1753-21gnome-shell20:20:323
3420353994522,19cyclictest1753-21gnome-shell20:20:323
342034299458,33cyclictest1753-21gnome-shell21:58:500
3420342994517,22cyclictest3492897-21sh22:30:290
3420342994514,27cyclictest1825-21JS14:31:440
3420342994514,27cyclictest1825-21JS14:31:440
3420342994513,28cyclictest3477600-21seq21:45:350
162399450,33rtkit-daemon2197014-21PK-Backend21:19:122
162399450,31rtkit-daemon1753-21gnome-shell22:58:012
342035399447,13cyclictest42-21ksoftirqd/319:50:293
3420353994419,10cyclictest42-21ksoftirqd/320:35:213
3420353994419,10cyclictest42-21ksoftirqd/320:35:213
342035399441,6cyclictest2197014-21PK-Backend20:50:013
342035399441,17cyclictest42-21ksoftirqd/323:58:193
3420353994410,19cyclictest42-21ksoftirqd/320:56:143
342034299442,37cyclictest3444413-21/usr/sbin/munin20:15:140
3420342994415,24cyclictest3504537-21cstates23:05:190
3420342994415,24cyclictest3504537-21cstates23:05:180
3420342994414,26cyclictest3450085-21latency20:30:220
3420342994412,28cyclictest3457402-21ls20:50:330
3420342994411,28cyclictest3463438-21sh21:10:140
162399440,33rtkit-daemon2197014-21PK-Backend21:13:152
162399440,32rtkit-daemon2197014-21PK-Backend20:34:022
342035399439,18cyclictest42-21ksoftirqd/323:49:133
342035399439,18cyclictest42-21ksoftirqd/323:49:123
342035399436,21cyclictest42-21ksoftirqd/323:34:353
342035399436,15cyclictest3485211-21taskset22:10:133
342035399436,13cyclictest0-21swapper/321:30:143
3420353994322,8cyclictest42-21ksoftirqd/321:10:033
3420353994321,8cyclictest42-21ksoftirqd/322:26:023
3420353994321,8cyclictest42-21ksoftirqd/322:26:023
3420353994315,25cyclictest42-21ksoftirqd/319:40:513
342035199439,15cyclictest1753-21gnome-shell23:50:142
342035199439,15cyclictest1753-21gnome-shell23:50:132
342034299432,36cyclictest1753-21gnome-shell21:35:170
342035399427,19cyclictest42-21ksoftirqd/319:47:033
3420353994220,8cyclictest42-21ksoftirqd/320:33:123
342035399421,26cyclictest2197014-21PK-Backend22:00:183
342035399421,26cyclictest2197014-21PK-Backend22:00:183
342035399421,17cyclictest42-21ksoftirqd/300:25:203
342035399421,15cyclictest2197014-21PK-Backend00:07:313
3420351994214,7cyclictest141rcu_preempt21:53:542
3420351994211,6cyclictest141rcu_preempt23:24:372
162399420,34rtkit-daemon2197014-21PK-Backend00:26:172
34495222410,8sleep22197014-21PK-Backend20:30:022
342035399417,12cyclictest42-21ksoftirqd/321:15:173
342035399415,20cyclictest42-21ksoftirqd/321:06:373
342035399414,13cyclictest42-21ksoftirqd/322:22:033
3420353994120,8cyclictest42-21ksoftirqd/323:02:243
3420353994119,19cyclictest391irq_work/322:17:453
3420353994117,8cyclictest42-21ksoftirqd/322:30:113
3420353994115,10cyclictest42-21ksoftirqd/300:00:043
3420353994114,14cyclictest42-21ksoftirqd/320:26:453
162399410,29rtkit-daemon2197014-21PK-Backend22:27:432
162399410,29rtkit-daemon2197014-21PK-Backend22:27:432
162399410,29rtkit-daemon2197014-21PK-Backend22:18:352
162399410,29rtkit-daemon2197014-21PK-Backend20:06:412
162399410,28rtkit-daemon34-21ksoftirqd/222:37:552
162399410,28rtkit-daemon2197014-21PK-Backend23:35:042
342035399408,15cyclictest2197014-21PK-Backend22:35:283
3420353994029,8cyclictest42-21ksoftirqd/321:50:123
3420353994017,8cyclictest42-21ksoftirqd/320:06:393
3420353994011,10cyclictest42-21ksoftirqd/300:20:013
342035199406,6cyclictest141rcu_preempt20:00:002
342035199406,12cyclictest17531gnome-shell22:54:032
342035199402,20cyclictest1753-21gnome-shell19:21:432
162399400,29rtkit-daemon2197014-21PK-Backend23:25:512
162399400,29rtkit-daemon2197014-21PK-Backend21:55:223
162399400,28rtkit-daemon2197014-21PK-Backend22:34:162
162399400,28rtkit-daemon2197014-21PK-Backend22:05:402
162399400,28rtkit-daemon2197014-21PK-Backend20:56:052
162399400,28rtkit-daemon2197014-21PK-Backend19:35:182
162399400,28rtkit-daemon2197014-21PK-Backend19:35:182
162399400,28rtkit-daemon1753-21gnome-shell20:24:412
162399400,28rtkit-daemon1753-21gnome-shell20:24:402
342035399395,19cyclictest42-21ksoftirqd/319:33:113
342035399394,17cyclictest42-21ksoftirqd/323:20:353
3420353993916,8cyclictest42-21ksoftirqd/320:18:153
342035199392,9cyclictest34-21ksoftirqd/200:39:332
342035199392,9cyclictest34-21ksoftirqd/200:39:322
342035199392,6cyclictest141rcu_preempt20:20:002
3420351993922,13cyclictest3480703-21ntpq21:55:292
342035199392,19cyclictest1753-21gnome-shell23:06:422
342035199392,19cyclictest1753-21gnome-shell23:06:422
342035199392,19cyclictest1753-21gnome-shell20:51:502
3420351993912,6cyclictest141rcu_preempt23:40:312
342035199391,21cyclictest1753-21gnome-shell21:05:212
162399390,5rtkit-daemon0-21swapper/321:44:043
162399390,5rtkit-daemon0-21swapper/321:44:043
162399390,28rtkit-daemon2197014-21PK-Backend22:42:082
162399390,28rtkit-daemon2197014-21PK-Backend19:28:392
162399390,27rtkit-daemon0-21swapper/221:49:532
162399390,26rtkit-daemon2197014-21PK-Backend20:12:012
162399390,26rtkit-daemon2197014-21PK-Backend19:30:052
342035199389,7cyclictest1938-21gsd-housekeepin00:06:372
3420351993824,10cyclictest3472595-21apt-key21:35:022
342035199382,32cyclictest1753-21gnome-shell22:02:262
342035199382,32cyclictest1753-21gnome-shell22:02:262
162399380,27rtkit-daemon2197014-21PK-Backend20:02:002
34729962370,7chrt2197014-21PK-Backend21:35:153
342035399371,12cyclictest42-21ksoftirqd/322:06:133
3420351993719,13cyclictest3455132-21df20:45:162
3420351993710,6cyclictest141rcu_preempt23:35:002
342035399361,8cyclictest2197014-21PK-Backend22:50:303
342035399361,15cyclictest2197014-21PK-Backend23:15:303
3420351993611,9cyclictest34-21ksoftirqd/221:40:172
3420351993611,9cyclictest34-21ksoftirqd/221:40:172
162399360,24rtkit-daemon0-21swapper/220:43:052
3420351993418,12cyclictest3524092-21df00:00:162
3420351993311,5cyclictest141rcu_preempt00:13:232
342035199322,13cyclictest1753-21gnome-shell22:20:222
3420351993112,15cyclictest1753-21gnome-shell00:18:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional