You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 12:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Dec 07, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
231345221030,7sleep3421ksoftirqd/323:50:223
231345221030,7sleep3421ksoftirqd/323:50:213
162399950,78rtkit-daemon0-21swapper/219:08:282
162399850,8rtkit-daemon869-21irqbalance00:34:140
162399810,7rtkit-daemon2197014-21PK-Backend22:01:393
162399810,7rtkit-daemon2197014-21PK-Backend22:01:393
162399740,9rtkit-daemon1775-21gvfs-afc-volume22:21:510
162399730,8rtkit-daemon2215920-21munin-node23:10:510
221348627240,25sleep10-21swapper/119:05:331
221360927139,26sleep00-21swapper/019:07:090
162399690,8rtkit-daemon856-21gmain22:34:510
162399690,8rtkit-daemon856-21gmain22:34:510
221370926848,13sleep30-21swapper/319:08:233
221398299629,33cyclictest1753-21gnome-shell21:19:253
2213982996219,20cyclictest1753-21gnome-shell22:14:373
2213979996227,19cyclictest34-21ksoftirqd/222:35:272
22537692610,7chrt2197014-21PK-Backend21:00:163
2213979996126,21cyclictest34-21ksoftirqd/200:00:222
162399610,10rtkit-daemon1622-21rtkit-daemon23:55:091
2213974996017,13cyclictest2246587-21ls20:40:161
2213982995921,12cyclictest42-21ksoftirqd/321:43:293
2213982995720,17cyclictest1753-21gnome-shell22:55:113
2213982995720,11cyclictest42-21ksoftirqd/323:32:493
221397999571,18cyclictest2245159-21/usr/sbin/munin20:35:182
2213974995713,17cyclictest2321204-21switchtime00:10:321
2213974995713,17cyclictest2321204-21switchtime00:10:321
162399570,9rtkit-daemon1622-21rtkit-daemon22:25:300
162399570,9rtkit-daemon1622-21rtkit-daemon00:39:290
22960712560,7sleep22197014-21PK-Backend23:00:182
2213974995626,6cyclictest2267465-21cpuspeed_turbos21:40:151
2213974995621,10cyclictest2319808-21mktemp00:10:011
2213974995621,10cyclictest2319808-21mktemp00:10:011
2213974995619,13cyclictest2298177-21ntp_states23:05:271
162399560,8rtkit-daemon1622-21rtkit-daemon00:26:101
2213974995521,7cyclictest2228631-21apt-config19:50:011
221397399557,9cyclictest1753-21gnome-shell20:30:230
162399550,9rtkit-daemon1622-21rtkit-daemon23:17:230
162399550,8rtkit-daemon1622-21rtkit-daemon22:39:010
221398299546,16cyclictest42-21ksoftirqd/319:50:353
2213982995417,19cyclictest1753-21gnome-shell21:59:353
221397999541,17cyclictest2234766-21/usr/sbin/munin20:05:202
2213974995420,8cyclictest1753-21gnome-shell22:12:581
2213974995417,14cyclictest2179-21gvfsd-metadata20:20:001
2213974995414,15cyclictest1753-21gnome-shell19:15:311
221397399549,11cyclictest2311885-21memory23:45:240
2213979995325,8cyclictest34-21ksoftirqd/221:45:132
2213974995323,11cyclictest26-21ksoftirqd/122:15:241
2213974995319,12cyclictest2225484-21wc19:40:161
2213974995317,9cyclictest2287882-21wc22:35:281
2213974995316,14cyclictest2230266-40apt-check19:51:181
221397399530,28cyclictest0-21swapper/000:15:160
162399530,9rtkit-daemon1622-21rtkit-daemon22:42:131
162399530,8rtkit-daemon1622-21rtkit-daemon23:25:331
2213974995221,6cyclictest2236070-21/usr/sbin/munin20:10:351
2213974995220,7cyclictest2329646-21sh00:35:201
2213974995218,11cyclictest2276002-21apt-get22:05:011
2213974995218,11cyclictest2276002-21apt-get22:05:011
2213974995217,11cyclictest2264335-21seq21:30:231
221397399528,15cyclictest2269939-21sed21:45:310
221397399528,15cyclictest2228201-21uptime19:45:330
162399520,9rtkit-daemon1622-21rtkit-daemon22:51:001
162399520,8rtkit-daemon1622-21rtkit-daemon23:33:190
162399520,8rtkit-daemon1622-21rtkit-daemon23:12:361
221398299518,19cyclictest1753-21gnome-shell19:10:163
221398299514,15cyclictest2291144-21needreboot22:45:223
2213982995127,20cyclictest1753-21gnome-shell20:00:243
2213982995126,8cyclictest42-21ksoftirqd/300:12:373
2213982995126,8cyclictest42-21ksoftirqd/300:12:373
2213979995130,8cyclictest34-21ksoftirqd/222:05:032
2213979995127,20cyclictest1753-21gnome-shell21:15:262
2213979995114,19cyclictest34-21ksoftirqd/219:29:012
2213979995112,19cyclictest34-21ksoftirqd/219:15:032
221397499516,25cyclictest2285527-21expr22:30:151
221397499516,25cyclictest2285527-21expr22:30:141
221397499514,18cyclictest1827-21JS14:31:441
2213974995124,6cyclictest64850irq/126-i91519:35:001
221397499512,14cyclictest2312307-21timerandwakeup23:45:341
2213974995114,11cyclictest2281909-21grep22:20:151
221397499511,15cyclictest2244116-21/usr/sbin/munin20:30:351
221397399518,9cyclictest2257820-21cat21:10:320
221397399518,22cyclictest2225675-21expr19:40:170
221397399517,11cyclictest1753-21gnome-shell23:54:550
221397399517,11cyclictest1753-21gnome-shell23:54:550
2213973995110,12cyclictest1753-21gnome-shell22:12:180
2213973995110,11cyclictest2321555-31apt-check00:12:300
2213973995110,11cyclictest2321555-31apt-check00:12:290
2213982995019,11cyclictest1753-21gnome-shell23:43:093
2213982995014,14cyclictest42-21ksoftirqd/320:20:163
2213979995027,19cyclictest1753-21gnome-shell22:50:202
2213979995020,12cyclictest34-21ksoftirqd/222:15:332
2213979995017,16cyclictest1753-21gnome-shell21:44:582
2213974995039,7cyclictest26-21ksoftirqd/123:43:171
2213974995018,11cyclictest2323959-21cut00:20:141
2213974995013,13cyclictest2254434-21cat21:00:321
221397399509,21cyclictest1753-21gnome-shell20:29:110
221397399509,20cyclictest1753-21gnome-shell22:47:040
221397399509,10cyclictest2254016-21memory21:00:240
221397399506,8cyclictest2273064-21cstates21:55:190
221397399506,18cyclictest1753-21gnome-shell20:24:520
221397399504,27cyclictest1753-21gnome-shell20:51:460
221397399501,28cyclictest2237179-21users20:10:360
2213973995010,20cyclictest1753-21gnome-shell23:07:450
2213973995010,20cyclictest1753-21gnome-shell19:10:220
221398299498,21cyclictest1753-21gnome-shell23:23:463
221398299498,19cyclictest1753-21gnome-shell00:01:573
221398299497,21cyclictest1753-21gnome-shell21:30:103
221398299492,23cyclictest1753-21gnome-shell20:09:213
221398299491,28cyclictest2197014-21PK-Backend22:42:553
221397999497,18cyclictest34-21ksoftirqd/219:45:202
2213979994926,19cyclictest1753-21gnome-shell21:55:352
221397999491,23cyclictest1753-21gnome-shell23:29:402
2213974994915,11cyclictest2291467-21timerandwakeup22:45:311
221397399499,11cyclictest1753-21gnome-shell19:17:090
221397399497,14cyclictest2317195-21grep00:00:240
221397399496,20cyclictest1753-21gnome-shell21:35:070
2213973994925,8cyclictest13-21ksoftirqd/022:05:350
2213973994925,18cyclictest2293813-21grep22:55:020
2213973994910,19cyclictest1753-21gnome-shell20:19:080
2213973994910,17cyclictest2256249-21users21:05:330
2213973994910,17cyclictest2256249-21users21:05:320
221398299489,24cyclictest0-21swapper/319:24:213
221398299486,16cyclictest42-21ksoftirqd/323:12:483
221398299485,27cyclictest2197014-21PK-Backend23:26:483
2213982994818,11cyclictest1753-21gnome-shell19:58:353
2213982994814,17cyclictest1753-21gnome-shell19:25:083
221397999489,21cyclictest34-21ksoftirqd/223:17:352
2213979994824,20cyclictest1753-21gnome-shell19:50:182
2213979994818,13cyclictest34-21ksoftirqd/219:58:082
2213979994817,6cyclictest141rcu_preempt20:40:482
2213979994816,9cyclictest34-21ksoftirqd/223:30:372
2213979994810,13cyclictest141rcu_preempt23:44:182
221397499482,25cyclictest2259713-21users21:15:331
2213974994819,8cyclictest26-21ksoftirqd/100:30:271
2213974994819,8cyclictest2262986-21switchtime21:25:301
221397399489,19cyclictest1753-21gnome-shell00:20:090
221397399489,13cyclictest1753-21gnome-shell20:45:320
221397399488,14cyclictest2326395-21python300:25:320
221397399488,10cyclictest3242253-21nautilus20:40:020
221397399486,21cyclictest1753-21gnome-shell21:25:210
2213973994816,16cyclictest0-21swapper/019:32:450
221397399481,27cyclictest1753-21gnome-shell00:09:020
221397399481,27cyclictest1753-21gnome-shell00:09:010
221397399480,29cyclictest2296563-21python323:00:290
221398299477,16cyclictest42-21ksoftirqd/319:19:023
221398299476,13cyclictest2197014-21PK-Backend23:03:073
221398299475,19cyclictest1753-21gnome-shell00:20:413
221398299474,9cyclictest42-21ksoftirqd/321:20:313
221398299471,23cyclictest1753-21gnome-shell23:19:313
221397999472,19cyclictest1753-21gnome-shell00:23:152
2213979994719,9cyclictest34-21ksoftirqd/222:27:042
2213979994719,6cyclictest141rcu_preempt23:20:322
2213979994718,7cyclictest141rcu_preempt23:10:252
2213979994718,6cyclictest141rcu_preempt21:50:282
2213979994714,9cyclictest34-21ksoftirqd/222:35:012
2213979994714,9cyclictest34-21ksoftirqd/222:35:002
221397999471,25cyclictest1753-21gnome-shell20:00:562
221397999471,19cyclictest1753-21gnome-shell19:31:242
221397999471,18cyclictest1753-21gnome-shell21:00:062
2213979994710,5cyclictest141rcu_preempt23:37:442
2213979994710,14cyclictest2197014-21PK-Backend20:20:212
221397499474,22cyclictest2295410-21cron22:59:591
221397499472,12cyclictest1753-21gnome-shell21:10:321
2213974994719,7cyclictest2250176-21cstates20:50:171
221397399479,18cyclictest1753-21gnome-shell19:22:160
221397399479,17cyclictest13-21ksoftirqd/022:18:210
221397399479,12cyclictest1753-21gnome-shell21:50:270
221397399475,16cyclictest1753-21gnome-shell21:40:360
221397399473,15cyclictest2310750-21taskset23:41:420
221398299468,11cyclictest42-21ksoftirqd/320:53:193
221398299467,19cyclictest1753-21gnome-shell22:37:153
221398299467,15cyclictest42-21ksoftirqd/322:28:143
221398299466,19cyclictest1753-21gnome-shell22:52:543
221398299465,17cyclictest42-21ksoftirqd/320:42:213
221398299464,17cyclictest1753-21gnome-shell20:36:083
2213982994637,6cyclictest411rcuc/319:47:503
2213982994623,19cyclictest1753-21gnome-shell23:05:183
221397999468,18cyclictest34-21ksoftirqd/222:10:342
221397999468,16cyclictest854-21avahi-daemon21:26:202
221397999466,18cyclictest34-21ksoftirqd/220:35:012
2213979994623,7cyclictest34-21ksoftirqd/221:12:112
2213979994618,10cyclictest34-21ksoftirqd/222:00:262
2213979994618,10cyclictest34-21ksoftirqd/222:00:252
2213979994615,10cyclictest34-21ksoftirqd/222:43:012
2213979994613,13cyclictest34-21ksoftirqd/219:21:352
221397999461,16cyclictest2320761-21uname00:10:232
221397999461,16cyclictest2320761-21uname00:10:222
2213974994619,7cyclictest2276749-21cstates22:05:161
221397399469,10cyclictest1753-21gnome-shell23:35:090
221397399466,14cyclictest2275415-21smartctl22:00:310
221397399466,14cyclictest2275415-21smartctl22:00:300
221397399466,11cyclictest2303137-21kernelversion23:20:230
221397399465,16cyclictest1753-21gnome-shell23:55:220
221397399465,15cyclictest1753-21gnome-shell19:51:010
221398299457,19cyclictest1753-21gnome-shell23:38:343
221398299456,16cyclictest42-21ksoftirqd/300:26:193
221398299455,18cyclictest1753-21gnome-shell19:33:013
221398299455,16cyclictest42-21ksoftirqd/300:06:473
221398299455,16cyclictest42-21ksoftirqd/300:06:463
221398299454,17cyclictest1753-21gnome-shell20:57:503
221397999459,20cyclictest34-21ksoftirqd/220:15:172
221397999459,16cyclictest34-21ksoftirqd/220:55:292
221397999457,20cyclictest34-21ksoftirqd/219:35:212
2213979994536,6cyclictest331rcuc/223:05:522
221397999452,25cyclictest1753-21gnome-shell22:59:002
2213979994518,7cyclictest34-21ksoftirqd/200:19:582
2213979994515,5cyclictest2197014-21PK-Backend00:35:192
2213979994510,19cyclictest34-21ksoftirqd/200:05:232
2213979994510,19cyclictest34-21ksoftirqd/200:05:232
221397499452,13cyclictest2301270-21df_inode23:15:191
221397499451,29cyclictest231irq_work/100:02:441
221397399459,10cyclictest1753-21gnome-shell19:26:180
221397399455,16cyclictest2246749-21cat20:40:180
2213973994511,10cyclictest1753-21gnome-shell21:20:270
162399450,37rtkit-daemon1753-21gnome-shell23:50:521
162399450,37rtkit-daemon1753-21gnome-shell23:50:511
22660772440,7chrt2197014-21PK-Backend21:35:212
221398299442,19cyclictest2282914-21taskset22:20:563
2213982994415,12cyclictest141rcu_preempt20:32:553
221397999447,19cyclictest34-21ksoftirqd/220:29:432
2213979994418,8cyclictest1753-21gnome-shell21:31:542
2213979994417,6cyclictest141rcu_preempt22:45:062
2213979994411,15cyclictest34-21ksoftirqd/223:57:572
2213979994410,15cyclictest34-21ksoftirqd/222:23:352
2213979994410,15cyclictest34-21ksoftirqd/200:30:222
2213974994418,9cyclictest26-21ksoftirqd/121:50:121
2213974994415,25cyclictest1-21systemd23:32:061
2213974994411,7cyclictest0-21swapper/119:35:231
221397399448,10cyclictest1753-21gnome-shell22:55:530
221397399447,13cyclictest2231973-21users19:55:320
221397399447,11cyclictest1753-21gnome-shell19:36:190
221397399446,14cyclictest2305639-21chrt23:27:520
221397399445,14cyclictest2259592-21expr21:15:300
221397399445,12cyclictest2234332-21/usr/sbin/munin20:05:290
2213973994421,18cyclictest2252077-21diskstats20:55:190
221398299437,16cyclictest411rcuc/319:36:113
221398299437,13cyclictest42-21ksoftirqd/321:10:323
221398299435,15cyclictest42-21ksoftirqd/320:15:153
221398299434,8cyclictest42-21ksoftirqd/322:15:153
221398299434,13cyclictest42-21ksoftirqd/300:30:153
221398299433,13cyclictest42-21ksoftirqd/321:05:373
221398299433,13cyclictest42-21ksoftirqd/321:05:373
2213982994315,7cyclictest42-21ksoftirqd/321:51:523
221398299431,17cyclictest42-21ksoftirqd/323:56:063
221398299430,20cyclictest1753-21gnome-shell21:25:193
221397999439,18cyclictest34-21ksoftirqd/221:05:312
221397999439,18cyclictest34-21ksoftirqd/221:05:312
221397999439,18cyclictest34-21ksoftirqd/220:50:272
221397999437,20cyclictest34-21ksoftirqd/220:10:262
2213979994335,6cyclictest331rcuc/200:26:372
2213979994315,13cyclictest34-21ksoftirqd/223:50:032
2213979994315,13cyclictest34-21ksoftirqd/223:50:032
2213979994313,7cyclictest34-21ksoftirqd/219:44:172
2213979994312,13cyclictest141rcu_preempt19:13:062
2213974994314,10cyclictest64850irq/126-i91519:25:131
221397399439,10cyclictest2233023-21hddtemp_smartct20:00:200
221397399436,10cyclictest2289927-21pool-udisksd22:41:550
221398299428,14cyclictest42-21ksoftirqd/300:39:583
221398299426,8cyclictest2285734-21irqstats22:30:213
221398299426,8cyclictest2285734-21irqstats22:30:213
221398299425,16cyclictest42-21ksoftirqd/322:05:123
2213982994233,6cyclictest411rcuc/300:17:393
2213982994213,7cyclictest42-21ksoftirqd/320:46:513
221397999427,17cyclictest2197014-21PK-Backend21:20:122
221397999425,16cyclictest34-21ksoftirqd/223:45:022
2213973994210,9cyclictest2264586-21ntp_states21:30:280
22196862412,20sleep126-21ksoftirqd/119:22:261
221398299417,15cyclictest42-21ksoftirqd/321:38:433
221398299417,15cyclictest42-21ksoftirqd/319:41:033
221398299415,9cyclictest2197014-21PK-Backend21:46:563
221398299413,7cyclictest42-21ksoftirqd/323:45:073
2213979994112,7cyclictest34-21ksoftirqd/220:45:172
2213974994123,13cyclictest2266433-21sensors_temp21:35:281
221397499412,15cyclictest1753-21gnome-shell21:57:181
221398299405,16cyclictest42-21ksoftirqd/320:27:003
2213974994030,6cyclictest2232386-21unin-run20:00:011
221397499402,33cyclictest1753-21gnome-shell20:47:171
221398299394,16cyclictest2197014-21PK-Backend20:10:093
2213974993925,10cyclictest26-21ksoftirqd/121:48:121
2213974993925,10cyclictest2309098-21sshd23:37:461
2213974993827,7cyclictest2215959-21cat19:15:021
2213974993822,12cyclictest2284005-21/usr/sbin/munin22:25:241
221397499381,10cyclictest0-21swapper/120:20:211
2213974993724,9cyclictest1827-21JS14:31:441
221397499372,11cyclictest2322414-21df_inode00:15:171
2213974993716,15cyclictest1-21systemd20:10:011
2213974993520,11cyclictest2260358-21basename21:20:111
162399350,22rtkit-daemon26-21ksoftirqd/121:09:551
162399350,22rtkit-daemon26-21ksoftirqd/121:09:551
2213974993414,16cyclictest2241281-21find20:25:121
2213974993318,11cyclictest2962348-21rs:main1
2213974993318,11cyclictest1-21systemd20:40:011
221397499329,19cyclictest1617-10pipewire-media-23:00:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional