You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 05, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75087123060,7sleep00-21swapper/019:05:230
81723921160,13sleep0161rcuc/022:15:320
1623991130,9rtkit-daemon1622-21rtkit-daemon20:50:111
83268521030,5sleep20-21swapper/223:00:212
79736521030,11sleep2331rcuc/221:20:162
8117062930,6sleep20-21swapper/222:00:242
162399850,73rtkit-daemon0-21swapper/219:09:052
162399830,9rtkit-daemon1506-21systemd22:43:180
75159027645,22sleep30-21swapper/319:09:053
75018027639,9sleep10-21swapper/119:05:041
8531872620,11sleep0161rcuc/000:00:010
162399560,9rtkit-daemon1622-21rtkit-daemon19:26:570
162399560,8rtkit-daemon1622-21rtkit-daemon20:04:500
7794682540,6sleep0778893-21kworker/0:220:25:160
162399540,9rtkit-daemon1622-21rtkit-daemon19:53:591
162399530,9rtkit-daemon1622-21rtkit-daemon19:39:010
162399530,8rtkit-daemon1622-21rtkit-daemon20:50:050
8669882520,7sleep20-21swapper/200:35:322
751816995225,23cyclictest785812-21pool-org.gnome.20:45:012
751816995225,23cyclictest785812-21pool-org.gnome.20:45:002
162399520,9rtkit-daemon1622-21rtkit-daemon22:54:440
162399520,8rtkit-daemon1622-21rtkit-daemon19:34:570
75181799514,26cyclictest1753-21gnome-shell23:52:513
75181799511,17cyclictest854-21avahi-daemon23:43:573
75181799502,11cyclictest787265-21chrt20:46:573
751817995014,19cyclictest779118-21apt20:25:133
751817994925,20cyclictest772630-21ntp_states20:05:283
751806994930,7cyclictest13-21ksoftirqd/021:21:270
751806994929,7cyclictest13-21ksoftirqd/000:21:150
162399490,35rtkit-daemon2197014-21PK-Backend21:05:082
75181799485,19cyclictest1753-21gnome-shell19:16:103
751817994822,21cyclictest838074-40lsb_release23:15:243
75181799481,11cyclictest805095-21sed21:40:383
8606502470,6sleep10-21swapper/100:20:141
7613592470,4sleep10-21swapper/119:35:011
75181799478,14cyclictest837270-21cat23:15:023
75181799478,11cyclictest821189-31apt-check22:27:293
75181799478,11cyclictest821189-31apt-check22:27:293
75181799477,12cyclictest1753-21gnome-shell19:23:123
75181799476,20cyclictest766596-21latency_hist19:50:013
75181799471,21cyclictest1753-21gnome-shell21:30:123
751806994726,8cyclictest13-21ksoftirqd/019:10:290
751806994723,14cyclictest13-21ksoftirqd/020:21:230
162399470,36rtkit-daemon1753-21gnome-shell00:33:592
8432002460,5chrt0-21swapper/123:30:231
75181799465,10cyclictest1753-21gnome-shell00:15:013
75181799462,22cyclictest1753-21gnome-shell21:50:043
751812994620,22cyclictest776794-21cron20:20:001
8661612450,5sleep02197014-21PK-Backend00:35:150
75181799457,12cyclictest1753-21gnome-shell00:06:043
75181799456,17cyclictest834257-21cat23:05:203
75181799456,13cyclictest1753-21gnome-shell00:24:043
75181799456,11cyclictest807260-21munin-run21:50:013
75181799456,10cyclictest769906-21munin-run20:00:023
75181799455,15cyclictest42-21ksoftirqd/320:15:323
75181799454,16cyclictest438-20systemd-journal22:10:013
75181799454,16cyclictest438-20systemd-journal22:10:003
75181799451,13cyclictest2197014-21PK-Backend19:33:343
751816994531,10cyclictest764629-21sshd19:44:412
751816994513,6cyclictest8629722wc00:25:242
75181299457,21cyclictest2197014-21PK-Backend21:39:591
751812994516,13cyclictest26-21ksoftirqd/123:10:311
751806994525,7cyclictest13-21ksoftirqd/022:35:140
751806994524,7cyclictest13-21ksoftirqd/020:10:260
751806994523,13cyclictest13-21ksoftirqd/023:05:020
751806994523,13cyclictest13-21ksoftirqd/020:30:290
751806994521,15cyclictest13-21ksoftirqd/021:40:020
162399450,31rtkit-daemon1753-21gnome-shell21:59:042
75181799448,19cyclictest1753-21gnome-shell20:14:323
75181799448,18cyclictest1753-21gnome-shell22:42:353
75181799447,15cyclictest830458-21/usr/sbin/munin22:55:153
75181799445,13cyclictest824301-21cat22:35:323
75181799441,22cyclictest851890-21cstates23:55:153
75181799441,22cyclictest1753-21gnome-shell00:27:373
75181799441,13cyclictest42-21ksoftirqd/321:10:143
751816994412,13cyclictest141rcu_preempt20:15:492
751812994416,24cyclictest804902-21sensors_temp21:40:351
751806994431,8cyclictest827587-21grep22:45:250
751806994425,7cyclictest13-21ksoftirqd/023:25:150
751806994425,7cyclictest13-21ksoftirqd/023:15:220
751806994424,7cyclictest13-21ksoftirqd/020:55:200
751806994423,7cyclictest13-21ksoftirqd/022:00:290
751806994422,13cyclictest13-21ksoftirqd/022:25:320
751806994422,13cyclictest13-21ksoftirqd/022:25:320
751806994422,13cyclictest13-21ksoftirqd/021:40:360
751806994419,21cyclictest1753-21gnome-shell21:05:250
8153032430,7chrt2197014-21PK-Backend22:10:301
75181799439,14cyclictest42-21ksoftirqd/322:45:283
75181799437,12cyclictest759208-40lsb_release19:27:203
75181799437,10cyclictest818979-21users22:20:323
75181799437,10cyclictest818979-21users22:20:323
75181799436,13cyclictest1753-21gnome-shell21:25:263
75181799436,13cyclictest1753-21gnome-shell21:25:253
75181799436,10cyclictest843534-21ntp_states23:30:293
75181799436,10cyclictest803382-21users21:35:323
75181799435,18cyclictest1753-21gnome-shell20:59:223
75181799435,10cyclictest752372-21df_inode19:10:173
751817994324,14cyclictest859008-21df00:15:173
75181799432,23cyclictest1753-21gnome-shell21:04:243
75181799432,22cyclictest1753-21gnome-shell20:02:423
751816994324,14cyclictest816524-21df22:15:152
751812994327,12cyclictest790628-21sshd20:55:411
751812994318,6cyclictest26-21ksoftirqd/121:01:571
751806994325,7cyclictest13-21ksoftirqd/020:44:340
751806994325,7cyclictest13-21ksoftirqd/020:44:340
751806994325,6cyclictest13-21ksoftirqd/020:15:330
751806994324,7cyclictest13-21ksoftirqd/023:45:000
751806994324,7cyclictest13-21ksoftirqd/022:05:340
751806994324,7cyclictest13-21ksoftirqd/022:05:330
751806994324,7cyclictest13-21ksoftirqd/021:25:300
751806994324,7cyclictest13-21ksoftirqd/021:25:290
751806994324,6cyclictest13-21ksoftirqd/023:45:150
751806994323,7cyclictest13-21ksoftirqd/022:23:000
751806994323,7cyclictest13-21ksoftirqd/022:23:000
751806994323,7cyclictest13-21ksoftirqd/020:45:290
751806994321,13cyclictest13-21ksoftirqd/000:30:140
75181799429,17cyclictest42-21ksoftirqd/322:50:223
75181799428,16cyclictest842515-21munin-run23:30:013
75181799425,9cyclictest866291-21hddtemp_smartct00:35:193
75181799425,15cyclictest42-21ksoftirqd/321:07:153
75181799424,16cyclictest864132-21acpi00:30:113
75181799423,14cyclictest816160-21apt22:15:123
751817994230,8cyclictest845175-21tail23:35:233
751817994226,12cyclictest768099-21sshd19:54:413
75181799421,24cyclictest1753-21gnome-shell23:24:443
75181799421,23cyclictest1753-21gnome-shell20:54:573
75181799421,23cyclictest1753-21gnome-shell20:23:313
751817994211,11cyclictest42-21ksoftirqd/321:16:023
75181699425,9cyclictest141rcu_preempt19:24:002
75181699424,20cyclictest1753-21gnome-shell20:25:312
75181699422,22cyclictest761970-21kernelversion19:35:242
751816994213,12cyclictest141rcu_preempt21:29:302
751816994213,12cyclictest141rcu_preempt21:29:292
75181699421,22cyclictest846599-21expr23:40:152
751816994212,13cyclictest141rcu_preempt23:58:422
751816994212,12cyclictest141rcu_preempt19:55:142
751816994212,11cyclictest141rcu_preempt22:37:042
751816994211,12cyclictest141rcu_preempt23:53:222
751816994210,12cyclictest141rcu_preempt21:14:032
751816994210,11cyclictest141rcu_preempt20:10:372
751812994215,6cyclictest26-21ksoftirqd/123:40:121
75181299421,23cyclictest821404-21latency_hist22:30:021
75181299421,23cyclictest821404-21latency_hist22:30:021
751806994224,6cyclictest13-21ksoftirqd/021:51:050
751806994224,6cyclictest13-21ksoftirqd/021:30:290
751806994224,6cyclictest13-21ksoftirqd/021:15:300
751806994223,7cyclictest13-21ksoftirqd/023:05:430
751806994223,7cyclictest13-21ksoftirqd/021:10:280
751806994220,14cyclictest13-21ksoftirqd/022:00:020
162399420,31rtkit-daemon1753-21gnome-shell22:27:572
162399420,31rtkit-daemon1753-21gnome-shell22:27:572
8403012410,8chrt0-21swapper/223:20:302
75181799416,11cyclictest854-21avahi-daemon21:25:013
75181799415,12cyclictest833252-21taskset23:01:343
75181799415,10cyclictest809711-21irqstats21:55:223
75181799413,22cyclictest814959-21cat22:10:223
75181799411,16cyclictest8538172df_inode00:00:173
75181699419,12cyclictest141rcu_preempt19:46:252
751816994125,12cyclictest759455-21cron19:29:592
751816994113,5cyclictest141rcu_preempt19:10:252
751816994112,5cyclictest141rcu_preempt00:07:382
751816994112,11cyclictest141rcu_preempt20:46:392
75181699411,20cyclictest0-21swapper/223:15:352
751812994124,13cyclictest853222-21cups-browsed00:00:031
751812994117,20cyclictest822106-21cat22:30:231
751812994116,6cyclictest26-21ksoftirqd/123:55:191
75181299411,10cyclictest0-21swapper/100:35:271
75180699414,10cyclictest2197014-21PK-Backend23:35:250
751806994124,5cyclictest13-21ksoftirqd/021:01:530
751806994123,7cyclictest13-21ksoftirqd/000:00:280
751806994122,8cyclictest13-21ksoftirqd/020:07:240
751806994118,14cyclictest13-21ksoftirqd/000:11:450
75181799406,16cyclictest2197014-21PK-Backend19:42:453
75181799405,18cyclictest1753-21gnome-shell23:46:113
75181799401,9cyclictest2197014-21PK-Backend22:31:593
75181699409,12cyclictest141rcu_preempt23:45:212
75181699409,10cyclictest1753-21gnome-shell23:35:272
75181699408,9cyclictest141rcu_preempt00:01:572
75181699408,10cyclictest141rcu_preempt23:34:562
751816994025,11cyclictest795751-21/usr/sbin/munin21:15:192
751816994024,11cyclictest34-21ksoftirqd/223:10:142
751816994022,13cyclictest438-20systemd-journal23:07:102
751816994012,12cyclictest141rcu_preempt22:49:552
751816994011,5cyclictest141rcu_preempt22:20:282
751816994011,5cyclictest141rcu_preempt22:20:282
751816994011,12cyclictest141rcu_preempt21:46:392
751816994011,11cyclictest141rcu_preempt20:32:012
751816994011,11cyclictest141rcu_preempt19:17:192
751816994010,11cyclictest141rcu_preempt19:31:332
751812994023,12cyclictest438-20systemd-journal19:15:001
751812994013,11cyclictest778081-21cat20:20:291
751812994011,15cyclictest26-21ksoftirqd/119:20:151
751806994023,6cyclictest13-21ksoftirqd/019:59:200
751806994023,12cyclictest836811-21sshd23:12:400
751806994018,13cyclictest13-21ksoftirqd/000:15:230
162399400,30rtkit-daemon1753-21gnome-shell22:14:192
75181799397,10cyclictest762561-21users19:35:353
75181699398,12cyclictest141rcu_preempt20:50:372
75181699398,10cyclictest141rcu_preempt00:15:022
751816993922,12cyclictest777468-21grep20:20:172
751816993920,14cyclictest822678-21cat22:30:332
751816993914,6cyclictest141rcu_preempt20:05:572
751816993910,11cyclictest141rcu_preempt22:41:592
75181299399,13cyclictest141rcu_preempt23:50:081
75181299397,18cyclictest26-21ksoftirqd/120:28:401
751812993925,10cyclictest817433-21sshd22:17:421
751812993923,12cyclictest855-21dbus-daemon22:21:551
751812993923,12cyclictest855-21dbus-daemon22:21:551
751812993916,19cyclictest837968-40lsb_release23:15:231
751812993914,6cyclictest26-21ksoftirqd/119:36:141
751806993924,11cyclictest13-21ksoftirqd/020:35:130
75180699391,9cyclictest141rcu_preempt23:53:030
75180699391,4cyclictest141rcu_preempt00:25:270
751806993914,20cyclictest1-21systemd22:31:420
75181799386,15cyclictest2197014-21PK-Backend20:42:323
75181799386,15cyclictest2197014-21PK-Backend20:42:313
751817993822,12cyclictest783587-21sed20:35:303
75181799381,21cyclictest1753-21gnome-shell20:34:583
75181699386,9cyclictest141rcu_preempt20:01:332
75181699386,11cyclictest141rcu_preempt22:53:012
75181699386,10cyclictest141rcu_preempt22:55:272
75181699386,10cyclictest141rcu_preempt21:30:062
75181699385,10cyclictest141rcu_preempt21:40:252
751816993820,13cyclictest438-20systemd-journal00:25:002
751816993820,13cyclictest3242253-21nautilus22:10:022
751816993820,13cyclictest3242253-21nautilus22:10:022
751816993813,6cyclictest141rcu_preempt00:12:012
751816993812,6cyclictest141rcu_preempt21:51:132
75181699381,12cyclictest767736-21sed19:50:302
751816993811,11cyclictest141rcu_preempt20:55:162
751816993811,10cyclictest141rcu_preempt21:01:012
75181299388,13cyclictest26-21ksoftirqd/119:45:161
75181299386,15cyclictest793395-21kworker/1:121:53:561
75181299385,19cyclictest141rcu_preempt22:51:371
75181299383,19cyclictest848358-21df_inode23:45:161
75181299382,9cyclictest141rcu_preempt00:07:211
75181299382,32cyclictest1753-21gnome-shell20:47:521
751812993823,11cyclictest828323-21cron22:50:011
75181299382,19cyclictest1753-21gnome-shell23:21:551
751812993812,11cyclictest797291-21ls21:20:161
75180699382,32cyclictest1753-21gnome-shell19:51:520
751806993821,5cyclictest13-21ksoftirqd/019:40:140
162399380,25rtkit-daemon0-21swapper/223:26:122
75181799372,11cyclictest42-21ksoftirqd/322:01:463
751816993711,12cyclictest141rcu_preempt21:36:302
751812993722,11cyclictest784837-21cut20:40:261
751812993722,11cyclictest784837-21cut20:40:261
75181299372,19cyclictest1753-21gnome-shell20:33:111
751812993721,12cyclictest809961-21ntpq21:55:261
751812993721,12cyclictest769279-21sensors_temp19:55:311
75181299371,9cyclictest2197014-21PK-Backend21:30:531
751812993714,6cyclictest26-21ksoftirqd/122:55:191
751806993725,7cyclictest13-21ksoftirqd/023:20:290
75180699371,32cyclictest2197014-21PK-Backend19:16:390
751816993620,12cyclictest782693-21cat20:35:142
751812993626,6cyclictest1753-21gnome-shell20:15:001
75181299362,19cyclictest759220-40lsb_release19:27:221
75181299362,19cyclictest1753-21gnome-shell21:07:451
75181299362,18cyclictest811705-21ntp_states22:00:251
751812993621,11cyclictest770911-21awk20:00:281
75181299361,5cyclictest141rcu_preempt00:20:021
751812993614,6cyclictest26-21ksoftirqd/122:06:421
751812993614,6cyclictest26-21ksoftirqd/122:06:411
75181299361,19cyclictest141rcu_preempt00:35:021
75180699363,7cyclictest141rcu_preempt23:31:340
751812993519,12cyclictest800278-21cron21:30:011
751812993519,12cyclictest800278-21cron21:30:011
75181299351,18cyclictest764078-21python319:40:291
751812993511,12cyclictest26-21ksoftirqd/100:10:451
75180699356,4cyclictest141rcu_preempt21:48:390
751806993527,5cyclictest0-21swapper/019:24:400
751806993524,6cyclictest13-21ksoftirqd/022:55:080
751806993519,6cyclictest13-21ksoftirqd/000:05:110
75180699351,14cyclictest766415-21turbostat19:50:000
75181299347,8cyclictest782586-21apt20:35:131
75181299347,7cyclictest26-21ksoftirqd/121:45:291
75181299342,6cyclictest141rcu_preempt00:25:261
751812993423,8cyclictest1753-21gnome-shell20:10:001
75181299342,17cyclictest825121-21/usr/sbin/munin22:40:191
75181299342,10cyclictest1753-21gnome-shell23:00:271
751812993417,12cyclictest845602-21awk23:35:321
75181299341,13cyclictest841426-21df_inode23:25:171
751806993423,6cyclictest13-21ksoftirqd/022:10:290
75181299332,9cyclictest823993-21memory22:35:231
751812993321,9cyclictest64850irq/126-i91523:05:261
751812993318,11cyclictest796979-21wc21:20:011
751812993316,13cyclictest793727-21sort21:10:151
75181299311,13cyclictest754936-21awk19:15:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional