You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Fri May 02, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3224692991221,22cyclictest3312406-21taskset23:14:491
32246899910740,19cyclictest13-21ksoftirqd/022:26:520
33413122990,7sleep30-21swapper/300:35:163
32242102960,8sleep00-21swapper/019:05:290
322469299905,22cyclictest26-21ksoftirqd/123:00:251
322455429060,22sleep30-21swapper/319:09:483
322469699873,23cyclictest0-21swapper/222:48:092
3224692998717,15cyclictest26-21ksoftirqd/122:28:541
3224692998715,21cyclictest26-21ksoftirqd/123:16:341
3224692998619,51cyclictest26-21ksoftirqd/123:25:221
322469299854,60cyclictest3302403-21diskstats22:45:181
322469699842,21cyclictest0-21swapper/223:20:202
322469699832,20cyclictest0-21swapper/222:28:362
322469299828,15cyclictest26-21ksoftirqd/122:24:211
322469299824,19cyclictest26-21ksoftirqd/122:57:161
3224692998217,7cyclictest141rcu_preempt23:23:221
322469299821,15cyclictest3304566-21/usr/sbin/munin22:50:291
3224692998118,51cyclictest141rcu_preempt22:38:141
3224701998033,43cyclictest1753-21gnome-shell22:50:593
3224692998011,56cyclictest26-21ksoftirqd/122:30:311
3224701997962,5cyclictest42-21ksoftirqd/323:03:493
322469699792,8cyclictest0-21swapper/222:24:332
322469699792,19cyclictest0-21swapper/223:15:132
322469699792,18cyclictest0-21swapper/222:36:452
322469299798,18cyclictest26-21ksoftirqd/122:40:141
3224692997918,6cyclictest141rcu_preempt23:05:591
3224701997854,13cyclictest42-21ksoftirqd/322:45:203
3224701997829,45cyclictest1753-21gnome-shell23:13:133
322469699783,18cyclictest0-21swapper/222:51:372
322469699782,21cyclictest0-21swapper/222:55:292
322469699782,18cyclictest0-21swapper/223:10:382
322469699772,18cyclictest0-21swapper/223:25:272
322469699772,18cyclictest0-21swapper/222:41:002
3224689997769,5cyclictest2197014-21PK-Backend23:02:280
3224689997737,8cyclictest141rcu_preempt23:26:120
322469699762,15cyclictest141rcu_preempt23:09:212
322469699752,9cyclictest0-21swapper/222:32:192
3224689997538,8cyclictest141rcu_preempt22:55:480
3224701997464,5cyclictest42-21ksoftirqd/322:27:163
3224701997424,46cyclictest1753-21gnome-shell22:58:373
3224692997415,7cyclictest141rcu_preempt23:30:041
3224701997322,46cyclictest1753-21gnome-shell23:28:443
322470199725,6cyclictest141rcu_preempt23:22:353
322470199722,66cyclictest1753-21gnome-shell23:05:443
322469699722,15cyclictest0-21swapper/223:01:492
3224689997229,16cyclictest141rcu_preempt22:54:180
3224689997226,16cyclictest141rcu_preempt23:14:560
3224689997219,15cyclictest141rcu_preempt23:19:210
3224689997219,11cyclictest13-21ksoftirqd/022:50:020
3224689997217,43cyclictest13-21ksoftirqd/023:22:380
322426327238,9sleep20-21swapper/219:06:012
3224701997020,9cyclictest1753-21gnome-shell23:15:203
3224701997020,9cyclictest1753-21gnome-shell22:37:543
322468999709,52cyclictest13-21ksoftirqd/022:41:270
3224701996934,28cyclictest42-21ksoftirqd/322:30:263
3224689996820,7cyclictest0-21swapper/022:39:420
162399680,9rtkit-daemon1622-21rtkit-daemon00:38:482
162399680,11rtkit-daemon13-21ksoftirqd/021:39:181
3224689996738,18cyclictest13-21ksoftirqd/022:25:020
322468999662,21cyclictest1753-21gnome-shell22:30:270
322468999652,58cyclictest1753-21gnome-shell23:06:390
162399640,10rtkit-daemon1622-21rtkit-daemon21:44:023
322470199632,10cyclictest1753-21gnome-shell22:43:163
3224689996331,9cyclictest141rcu_preempt23:30:040
322469699620,57cyclictest1622-21rtkit-daemon21:39:462
322422426140,14sleep10-21swapper/119:05:301
322470199602,54cyclictest42-21ksoftirqd/322:22:373
322470199592,14cyclictest3317862-21taskset23:30:033
322469699571,51cyclictest1753-21gnome-shell23:30:082
162399570,9rtkit-daemon1622-21rtkit-daemon21:11:090
32269152560,7sleep22197014-21PK-Backend19:15:052
32269152560,7sleep22197014-21PK-Backend19:15:042
162399550,8rtkit-daemon1622-21rtkit-daemon22:02:362
3224701995423,21cyclictest42-21ksoftirqd/323:35:293
162399540,8rtkit-daemon1622-21rtkit-daemon21:47:382
3224701995020,21cyclictest42-21ksoftirqd/300:10:293
322469299500,46cyclictest1622-21rtkit-daemon21:10:011
3224689995025,15cyclictest13-21ksoftirqd/020:55:230
33336162490,6chrt2197014-21PK-Backend00:12:220
32382332490,6sleep32197014-21PK-Backend19:45:253
162399480,6rtkit-daemon2197014-21PK-Backend20:18:233
3224701994721,22cyclictest42-21ksoftirqd/323:50:333
3224701994721,18cyclictest42-21ksoftirqd/320:01:163
3224696994718,11cyclictest3266043-21seq21:00:192
322469699471,13cyclictest3237671-21grep19:45:012
3224689994727,15cyclictest13-21ksoftirqd/019:42:260
162399470,40rtkit-daemon1506-21systemd21:15:140
162399470,34rtkit-daemon1753-21gnome-shell21:05:313
3224701994620,21cyclictest42-21ksoftirqd/320:25:303
3224701994619,22cyclictest42-21ksoftirqd/321:45:293
3224692994627,14cyclictest438-20systemd-journal19:40:011
3224689994625,15cyclictest13-21ksoftirqd/019:20:400
3224689994613,11cyclictest141rcu_preempt20:49:090
32925922450,15sleep1251rcuc/122:15:321
32678062450,6sleep20-21swapper/221:05:202
32659092450,7chrt2197014-21PK-Backend21:00:193
3224701994525,14cyclictest438-20systemd-journal20:05:323
3224701994518,22cyclictest42-21ksoftirqd/323:45:243
3224701994511,4cyclictest141rcu_preempt20:10:303
3224696994514,13cyclictest34-21ksoftirqd/220:45:352
3224689994525,15cyclictest13-21ksoftirqd/019:35:220
3224689994523,7cyclictest13-21ksoftirqd/020:30:320
3224689994523,7cyclictest13-21ksoftirqd/020:30:310
3224689994522,13cyclictest13-21ksoftirqd/000:35:150
322468999451,9cyclictest13-21ksoftirqd/000:00:020
322468999451,9cyclictest13-21ksoftirqd/000:00:010
32466772440,6sleep1251rcuc/120:10:021
322470199449,5cyclictest141rcu_preempt00:30:013
3224701994419,7cyclictest1753-21gnome-shell00:03:083
3224692994429,12cyclictest3250342-21awk20:20:001
3224692994423,17cyclictest3253425-21users20:25:341
3224692994417,22cyclictest438-20systemd-journal20:35:021
3224692994417,22cyclictest438-20systemd-journal20:35:021
3224689994426,14cyclictest13-21ksoftirqd/020:00:140
3224689994425,6cyclictest13-21ksoftirqd/021:40:170
3224689994425,6cyclictest13-21ksoftirqd/020:55:010
3224689994424,16cyclictest13-21ksoftirqd/020:40:380
3224689994424,15cyclictest13-21ksoftirqd/023:45:230
3224689994422,13cyclictest13-21ksoftirqd/019:15:060
3224689994422,13cyclictest13-21ksoftirqd/019:15:050
162399440,31rtkit-daemon1753-21gnome-shell21:17:333
162399440,26rtkit-daemon2197014-21PK-Backend23:58:191
162399440,26rtkit-daemon2197014-21PK-Backend23:58:191
3224701994324,14cyclictest42-21ksoftirqd/321:55:153
322469699438,17cyclictest141rcu_preempt20:30:212
322469699438,17cyclictest141rcu_preempt20:30:202
322469699432,23cyclictest1753-21gnome-shell20:18:472
3224696994313,6cyclictest141rcu_preempt00:09:442
322469699431,12cyclictest3289985-21cat22:10:132
3224692994330,10cyclictest3261280-21sshd20:49:321
3224692994325,13cyclictest855-21dbus-daemon00:20:021
3224689994326,6cyclictest13-21ksoftirqd/023:45:020
3224689994326,12cyclictest3246563-21apt-get20:10:000
3224689994325,14cyclictest13-21ksoftirqd/000:00:140
3224689994324,15cyclictest13-21ksoftirqd/023:50:320
3224689994324,15cyclictest13-21ksoftirqd/021:05:210
3224689994323,16cyclictest13-21ksoftirqd/021:35:310
3224689994323,16cyclictest13-21ksoftirqd/020:12:250
3224689994323,16cyclictest13-21ksoftirqd/019:50:010
3224689994323,15cyclictest13-21ksoftirqd/021:00:280
3224689994322,8cyclictest13-21ksoftirqd/019:50:360
33391392420,17sleep013-21ksoftirqd/000:30:020
3224701994211,5cyclictest141rcu_preempt00:15:223
322469699424,9cyclictest141rcu_preempt00:14:192
3224696994227,11cyclictest3335578-21/usr/sbin/munin00:20:012
322469699422,10cyclictest1753-21gnome-shell20:20:282
3224696994217,7cyclictest1753-21gnome-shell21:40:192
3224696994212,8cyclictest34-21ksoftirqd/220:44:382
3224689994224,6cyclictest13-21ksoftirqd/020:30:010
3224689994224,14cyclictest13-21ksoftirqd/020:17:010
3224689994224,14cyclictest13-21ksoftirqd/019:55:330
3224689994224,14cyclictest13-21ksoftirqd/000:05:130
3224689994223,6cyclictest13-21ksoftirqd/022:20:020
3224689994223,15cyclictest13-21ksoftirqd/021:23:570
3224689994223,14cyclictest13-21ksoftirqd/021:30:360
3224689994222,7cyclictest13-21ksoftirqd/021:47:290
3224689994222,15cyclictest13-21ksoftirqd/019:25:130
3224689994210,8cyclictest32836502grep21:50:250
3224701994119,13cyclictest42-21ksoftirqd/323:40:283
3224701994115,22cyclictest1753-21gnome-shell22:00:223
322469699418,16cyclictest34-21ksoftirqd/223:35:312
3224696994126,11cyclictest3326583-21apt-get23:55:012
322469699412,21cyclictest3329668-21ntp_states00:00:272
3224696994117,6cyclictest141rcu_preempt23:40:372
3224696994112,12cyclictest3245918-21ps20:05:312
3224692994125,12cyclictest3322689-21sshd23:43:321
3224689994128,8cyclictest1753-21gnome-shell00:30:360
3224689994123,6cyclictest13-21ksoftirqd/022:05:010
3224689994123,14cyclictest13-21ksoftirqd/022:05:170
3224689994122,7cyclictest13-21ksoftirqd/020:35:280
3224689994122,6cyclictest13-21ksoftirqd/023:40:000
3224689994122,15cyclictest13-21ksoftirqd/020:20:170
322468999411,36cyclictest1753-21gnome-shell21:28:130
162399410,5rtkit-daemon0-21swapper/320:48:133
322470199402,33cyclictest1753-21gnome-shell00:23:073
3224701994017,18cyclictest42-21ksoftirqd/321:11:233
3224701994016,21cyclictest1753-21gnome-shell22:06:563
322470199401,34cyclictest3290270-21diskstats22:10:153
322470199401,16cyclictest141rcu_preempt19:55:313
322469699408,8cyclictest3242254-21gmain20:00:042
3224696994030,7cyclictest1069-21runrttasks00:27:442
3224696994022,13cyclictest3327789-21grep23:55:232
3224696994022,13cyclictest3327789-21grep23:55:232
322469699402,20cyclictest1753-21gnome-shell19:58:232
3224696994016,19cyclictest3235523-21latency_hist19:40:032
3224692994025,11cyclictest3232040-21cron19:29:591
3224692994024,12cyclictest3240616-21apt-get19:50:321
3224692994018,17cyclictest438-20systemd-journal21:55:021
3224692994015,7cyclictest3244304-21switchtime20:00:351
3224692994014,10cyclictest3332833-21if_err_enp2s000:10:221
3224692994013,11cyclictest3330893-21expr00:05:151
3224689994027,8cyclictest1753-21gnome-shell00:21:350
3224689994021,6cyclictest13-21ksoftirqd/019:35:010
322470199398,6cyclictest141rcu_preempt00:33:183
322470199394,10cyclictest42-21ksoftirqd/320:43:333
3224701993928,7cyclictest3229546-21needreboot19:20:253
3224701993922,12cyclictest3226999-21ntp_states19:15:063
3224701993922,12cyclictest3226999-21ntp_states19:15:053
322470199391,8cyclictest3240361-21cut19:50:303
322469699399,14cyclictest141rcu_preempt00:20:162
322469699396,14cyclictest141rcu_preempt21:56:292
322469699392,7cyclictest1753-21gnome-shell23:47:182
322469699392,20cyclictest3272283-21chrt21:19:342
3224696993913,8cyclictest34-21ksoftirqd/219:25:522
3224696993910,14cyclictest141rcu_preempt20:50:322
3224692993926,9cyclictest3326213-21users23:50:361
3224692993919,15cyclictest3265130-40apt-check20:57:171
3224692993917,18cyclictest3233795-21latency_hist19:35:021
3224689993922,6cyclictest13-21ksoftirqd/021:55:320
3224689993921,5cyclictest13-21ksoftirqd/019:11:150
322470199387,10cyclictest42-21ksoftirqd/319:37:553
322470199385,21cyclictest42-21ksoftirqd/321:30:263
322470199382,8cyclictest1753-21gnome-shell21:22:063
322470199381,14cyclictest3263504-21apt-config20:55:023
3224701993811,22cyclictest42-21ksoftirqd/321:35:313
322469699389,9cyclictest34-21ksoftirqd/222:16:352
322469699382,12cyclictest141rcu_preempt19:11:232
3224696993811,7cyclictest1753-21gnome-shell19:54:182
322469699381,13cyclictest1753-21gnome-shell20:14:472
322469299385,7cyclictest0-21swapper/121:45:271
322469299382,11cyclictest1753-21gnome-shell00:20:201
3224692993821,13cyclictest3269138-21apt21:10:161
3224692993820,13cyclictest3288949-21awk22:05:311
3224692993817,15cyclictest1-21systemd21:30:011
3224689993821,6cyclictest13-21ksoftirqd/022:10:180
322470199379,4cyclictest141rcu_preempt20:30:023
322470199379,4cyclictest141rcu_preempt20:30:023
322470199374,11cyclictest42-21ksoftirqd/323:58:173
322470199374,11cyclictest42-21ksoftirqd/323:58:163
3224701993711,6cyclictest141rcu_preempt00:08:193
322470199371,13cyclictest2197014-21PK-Backend20:20:293
322469699375,17cyclictest141rcu_preempt22:06:482
322469699372,9cyclictest1753-21gnome-shell20:57:282
322469699372,9cyclictest1753-21gnome-shell19:45:232
3224696993717,15cyclictest3229229-21df19:20:152
3224696993713,7cyclictest3275183-21processes21:25:312
322469699371,10cyclictest854-21avahi-daemon21:25:022
322469299372,7cyclictest1753-21gnome-shell20:37:031
3224692993724,9cyclictest26-21ksoftirqd/119:55:321
3224692993721,11cyclictest3320339-21ntp_kernel_err23:35:261
3224692993720,13cyclictest3339358-21cat00:30:141
3224692993720,13cyclictest3271769-21cat21:15:301
3224692993720,12cyclictest468-21systemd-udevd21:02:501
3224692993719,13cyclictest3242255-21gdbus20:45:001
3224692993715,18cyclictest3248066-21timerandwakeup20:10:341
3224692993714,6cyclictest141rcu_preempt20:55:031
322469299371,32cyclictest3286190-21awk22:00:001
3224692993713,20cyclictest3329431-21cut00:00:221
162399370,25rtkit-daemon0-21swapper/123:48:211
322470199363,11cyclictest141rcu_preempt19:25:403
3224701993623,10cyclictest42-21ksoftirqd/322:19:523
3224701993620,12cyclictest42-21ksoftirqd/319:13:263
3224701993618,14cyclictest3233071-21ntp_states19:30:273
3224701993611,5cyclictest141rcu_preempt21:00:003
322469699368,10cyclictest1753-21gnome-shell21:12:462
322469699362,9cyclictest1753-21gnome-shell21:50:162
322469699362,9cyclictest1753-21gnome-shell20:25:242
3224696993621,11cyclictest3257551-21cron20:39:592
322469699362,10cyclictest1753-21gnome-shell19:30:332
3224692993623,9cyclictest26-21ksoftirqd/100:35:121
3224689993618,6cyclictest13-21ksoftirqd/000:15:220
32252602350,8sleep10-21swapper/119:10:171
322470199356,9cyclictest3263478-21kworker/3:121:28:233
322470199352,10cyclictest0-21swapper/319:42:283
3224692993525,6cyclictest3277513-21sh21:35:001
3224692993522,9cyclictest26-21ksoftirqd/120:20:281
322469299352,10cyclictest3242253-21nautilus19:50:011
3224692993520,11cyclictest3337819-21expr00:25:151
3224692993513,18cyclictest438-20systemd-journal19:43:321
322469299351,15cyclictest3291362-21turbostat22:15:011
322470199348,6cyclictest141rcu_preempt20:37:253
3224701993422,8cyclictest1753-21gnome-shell21:50:433
322469699348,6cyclictest141rcu_preempt21:30:182
322469699342,17cyclictest1753-21gnome-shell00:34:062
3224692993419,11cyclictest3287928-21pluginstate22:05:011
3224692993419,11cyclictest3274274-21cut21:25:021
3224692993410,20cyclictest3227142-21cpu19:15:151
3224692993410,20cyclictest3227142-21cpu19:15:151
3224692993317,12cyclictest3280435-21cat21:40:331
322469299331,12cyclictest3229309-21if_enp2s019:20:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional